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I.  INTRODUCTION 


A.  RESEARCH  OBJECTIVE 

Development  of  new  and  improved  weapon  systems  requires  technology  to  expand  its 
boundaries.  The  Department  of  the  Navy  is  a  prime  requirer  of  these  state-of-the-art  (SOA) 
technological  advancements.  However,  the  budgetary  impacts  of  SOA  extensions  are  often 
greater  than  anticipated.  The  Navy  budget  development  process  does  not  appear  to  include 
an  adequate  methodology  for  determining  what  extending  an  existing  weapon  system 
"should  cost".  To  realistically  budget  for  SOA  extensions,  Department  of  the  Navy  budget 
analysts  should  better  understand  how  cost  estimating  for  SOA  extension  projects  is  cur¬ 
rently  accomplished  by  the  individual(s)  responsible  for  advancing  this  technology,  i.e.  the 
defense  industry.  Navy  budget  analysts  can  use  this  knowledge  to  test  the  accuracy  of  the 
various  cost  estimating  techniques.  The  purpose  of  this  research  was  to  gain  greater  insight 
into  how  one  major  corporation  involved  in  Navy  contracting  forecasts  or  predicts  the  de¬ 
velopment  costs  associated  with  extending  technology  beyond  current  bounds. 

B.  RESEARCH  QUESTIONS 

Given  the  preceding  objective,  the  following  primary  research  question  was  posed: 
What  methods  were  employed  by  a  major  Department  of  the  Navy  contractor  to  predict  the 
development  costs  associated  with  an  actual  state-of-the-art  (SOA)  extension  project? 

In  addition,  the  following  subsidiary  research  questions  were  considered  relevant  in 
thoroughly  addressing  the  primary  question: 

1 .  What  were  the  expected  costs,  actual  costs,  and  variances  experienced  during  the 
development  of  the  SOA  extension  project? 

2.  What  were  the  reasons  for  the  variances  experienced? 


3.  Which  development  cost  categories  had  the  most  effect  on  total  project  cost?,  i.e- 
what  were  the  "cost  drivers"? 

4.  What  organizational  design  or  structure  was  utilized  during  the  SOA  project;  and 
how  does  that  design  affect  cost  management? 

C.  SCOPE,  LIMITATIONS,  AND  ASSUMPTIONS 

1.  Scope 

The  research  focused  on  the  actual  cost  estimation  techniques  utilized  by  a  De¬ 
partment  of  the  Navy  contractor  in  the  development  of  an  SOA  extension  project.  No  effort 
was  made  to  consider  the  justification  or  fairness  of  profit  margins  or  return  on  invest¬ 
ments;  rather  emphasis  was  placed  on  determining  the  development  costs  associated  with 
the  SOA  extension.  In  addition,  the  study  did  not  attempt  to  construct  or  suggest  any  ana¬ 
lytical  models  for  predictive  or  explanatory  purposes.  The  basic  effort  was  aimed  at  pro¬ 
ducing  an  accurate  description  of  the  quantitative  and  qualitative  development  costing 
methods  employed  by  a  particular  defense  contractor.  There  has  been  no  prior  research  di¬ 
rectly  addressing  the  primary  research  question.  Consequently,  the  researcher  relied  upon 
personal  interviews  with  key  contractor  personnel  and  the  collection  and  examination  of 
cost  data  at  the  contractor  site.  The  results  of  this  study  should  provide  the  reader  with  a 
complete  understanding  of  the  actual  process  involved  and  results  achieved  by  a  specific 
contractor  on  an  actual  SOA  extension  project. 

2.  LimilatiQna 

The  study  is  limited  to  the  observations  and  conclusions  reached  by  the  re¬ 
searcher  on  an  actual  SOA  extension  project  conducted  by  a  Department  of  the  Navy  con¬ 
tractor.  An  inherent  limitation  of  this  study  is  that  direct  transferability  of  the  data,  findings, 
and  observations  to  other  applications  may  be  inappropriate.  The  actual  SOA  project  stud¬ 
ied  involved  the  design  ,  construction,  and  operational  testing  of  a  single  final  product. 
Production  costs  are  seen  as  incorporated  in  the  development  costs  and  are  therefore  not 
considered  separately. 
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3. 


The  presumption  throughout  this  research  effort  is  that  the  Department  of  the 
Navy  contractor  routinely  employs  a  method  of  estimating  development  costs  associated 
with  SOA  extension  projects.  No  overly  stringent  assumptions  were  made  concerning  the 
readers  in-depth  knowledge  of  cost  accounting,  quantitative  techniques  for  cost  estimation, 
or  the  acquisition/contracting  process  within  the  Department  of  Defense. 

D.  RESEARCH  METHODOLOGY 

The  research  methodology  utilized  in  this  effort  consisted  of  a  case  study  involving  the 
cost  estimating  processes  employed  by  Lockheed  Missiles  and  Space  Company  in  devel¬ 
opment  of  the  Control  System  Test  Vehicle  (CSTV)  project.  Personal  interviews  and  doc¬ 
umentation  provided  by  the  Naval  Plant  Representative  Officer  (NAVPRO)  ,  Lockheed 
Corporation,  Sunnyvale,  California,  and  the  Lockheed  Missiles  and  Space  Company,  pro¬ 
vided  the  primary  input  for  the  research  effort.  Additional  background  research  consisted  of 
examining  the  literature  base  thorough  the  Defense  Technical  Information  Center  (DTIC), 
the  Defense  Logistics  Studies  Information  Exchange  (DLSIE),  and  various  publications 
and  journals. 

E.  ORGANIZATION  OF  THE  REPORT 

Chapter  II  begins  with  a  discussion  of  relevant  definitions  and  concepts  found  in  the 
literature;  such  as,  what  constitutes  an  extension  of  the  current  state-of-the-art;  what  items 
are  categorized  as  development  costs;  and  the  differences  between  cost  growth  involving 
contract  modifications  and  true  cost  overruns  or  variances.  Chapter  II  continues  with  a  brief 
discussion  of  the  types  of  contracts  used  to  support  SOA  extension  projects.  Generic  cost 
estimation  techniques  and  methods  used  by  the  Department  of  the  Navy  and  the  defense 
manufacturing  industry  are  also  considered.  Chapter  III  introduces  the  Control  System  Test 
Vehicle  (CSTV)  itself  and  the  cost  estimating  methods  employed  by  LMSC  in  estimating 
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the  project's  development  costs.  Chapter  III  also  includes  a  description  of  the  orga¬ 
nizational  structure  and  working  relationships  formed  within  LMSC  to  complete  the  Con¬ 
trol  System  Test  Vehicle  (CSTV)  project.  Chapter  FV  presents  the  cost  data  collected.  Vari¬ 
ances  between  predicted  and  actual  costs  are  examined  along  with  the  causative  factors  in¬ 
volved.  Cost  categories  that  exhibited  significant  influence  over  total  project  cost  are  identi¬ 
fied  and  discussed  at  the  end  of  Chapter  FV.  Chapter  V  summarizes  the  principle  findings 
of  the  study,  conclusions  reached,  and  practical  recommendations  made.  In  addition,  areas 
for  further  research  are  suggested. 
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II.  BACKGROUND  REVIEW  AND  THEORETICAL 

FRAMEWORK 

A.  INTRODUCTION 

The  purpose  of  this  chapter  is  to  provide  the  reader  with  an  overall  perspective  of  the 
environment  in  which  the  research  was  conducted.  The  chapter  begins  by  examining  and 
defining  relevant  concepts  and  terminology  associated  with  technical  advancements  in  state- 
of-the-art  weapons  systems.  The  important  difference  between  cost  deviations  due  to 
modifications  in  the  basic  contract  and  true  cost  overruns  is  then  discussed.  What  follows 
is  a  description  of  a  key  project  management  tool  used  within  the  contracting  effort;  the 
Contract  Data  Requirements  List  (DD  Form  1423).  A  brief  look  at  the  major  types  of  gov¬ 
ernment  contracts  awarded  to  state-of-the-art  extension  projects  provides  the  reader  with  a 
general  understanding  of  the  incentives  and  policies  under  which  a  development  contract  is 
administered.  The  two  basic  approaches  currently  used  by  the  Depanment  of  the  Navy  to 
estimate  development  costs  are  then  introduced.  Finally,  the  most  widely  accepted  generic 
cost  estimating  techniques  currently  in  use  by  the  defense  industry  will  be  presented. 

B.  DEFINITIONS  AND  CONCEPTS 

1.  State-of-the-Art  (SOA)  Extension 

Before  any  discussion,  a  definition  of  what  constitutes  a  state-of-the-an  (SOA) 
extension,  must  be  established.  Although,  an  exact  definition  of  the  term  is  difficult  to  ob¬ 
tain,  synonymous  definitions  are  found  in  the  technical  literature.  James  R.  Bright  and 
Milton  E.  F.  Shoeman  describe  the  process  of  technological  innovation  as  follows: 
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The  process  of  technological  innovation  is  a  phase  intended  to  embrace  those  activities 
by  which  technical  knowledge  is  translated  into  a  physical  reality  and  is  used  on  a  scale 


having  substantial  societal  impact .  This  definition  includes  more  than  the  act  of  inven¬ 
tion.  It  includes  initiation  of  the  technical  idea  and  acquisition  of  na:essary  knowledge,  its 
transformation  into  useable  hardware  (or  a  process),  its  introduction  into  society,  and  its 
diffusion  and  adoption  to  the  point  where  its  impact  is  significant.  [  Ref.  1  :p.48  ] 

Perhaps  the  most  precise  definition  of  "state-of-the-art"  technology  and  techno¬ 
logical  advancement  encountered  by  the  researcher  was  given  by  G.N.  Dodson  and  C.A. 
Graves  of  General  Research  Corporation  as  follows; 

The  state-of-the-art  is  the  state  of  best  implemented  technology  reflected  in  the  most  re¬ 
cently  [applied]  physical  and  performance  characteristics.  Our  overall  hypothesis  con¬ 
cerning  the  advancement  in  the  state-of-the-art  is  more  precisely  stated  as  the  develop¬ 
mental  design  characteristics,  (or  related  to  the  advance  of  these  characteristics)  repre¬ 
sented  in  relation  to  the  best  that  has  been  previously  implemented.  [  Ref.  2;p.l  ] 

With  the  above  ideas  in  mind,  the  researcher  has  adopted  the  following  as  an  ac¬ 
ceptable  definition  of  state-of-the-art  extension;  advancements  in  applied  technology, 
through  the  unique  combination  of  known  and  newly  developed  technological  methods,  to 
produce  a  previously  nonexistent  product.  Work  currently  in  progress  on  the  Strategic  De¬ 
fense  Initiative  (SDI)  program  provides  numerous  examples  of  advancing  technology 
through  state-of-the-art  extensions.  The  Exoatmospheric  Reentry  Vehicle  Interception  Sys¬ 
tem  (ERIS)  program  '  underdevelopment  at  Lxickheed  Corporation  combines  technologies 
developed  during  previous  projects  (propulsion,  ordnance,  structural)  with  current  and  yet 
undeveloped  technology  (  advanced  avionics,  engineering  integration)  to  produce  a  cur¬ 
rently  non  existent  defensive  missile  system. 

2.  Development  Cost 

As  stated  in  Chapter  One,  the  researcher  has  limited  the  scope  of  the  study 
exclusively  to  considerations  of  development  costs.  It  may  now  be  helpful  to  define 


‘  The  ERIS  program  is  currently  being  developed  for  the  U.S.  Army  Strategic 
Defense  Command  (  USASDC)  at  an  estimated  cost  of  $500M.  The  ERIS  program  will 
consist  of  a  system  of  ground  launched  ,  non-nuclear  missiles,  whose  purpose  will  be  to 
intercept  and  destroy  incoming  ICBMs. 
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precisely  what  costs  are  included  within  this  category.  Development  efforts  can  be  viewed 
(within  the  Department  of  Defense)  as  a  subset  of  a  broader  category  of  activity  generally 
referred  to  as  Research,  Development,  Test,  and  Evaluation  (RDT&E).  According  to  the 
Department  of  the  Navy  Budget  Guidarice  Manual,  the  overall  function  of  RDT&E  is  to 
provide  the  capabilities  needed  to  most  effectively  carry  out  the  tasks  required  to  success¬ 
fully  complete  the  mission  of  the  Navy  [  Ref.  3  ].  Moreover,  RDT&E  programs  should  fo¬ 
cus  on  (1)  determining  what  technologies  are  possible,  and  (2)  applying  what  is  technically 
possible,  to  develop  workable  solutions  that  satisfy  mission  requirements  [  Ref.  3  ].  The 
costs  associated  with  these  objectives  are  categorized  as  RTD&E  costs.  For  the  purpose  of 
this  study  development  costs  will  refer  to  all  costs  that  are  associated  with  the  design  and 
testing  of  a  new  operational  capability  ^  . 
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In  a  subsequent  chapter,  frequent  reference  will  be  made  to  the  terms  "cost 
growth"  and  "cost  overrun"  (or  variances).  There  is  often  confusion  regarding  these  two 
concepts;  and  therefore  a  tendency  to  erroneously  use  the  two  terms  interchangeably.  Cost 
growth  refers  to  change  in  current  cost  estimates  over  a  previously  established  base  figure. 
Therefore,  changes  in  the  total  estimated  cost  of  a  program  should  correctly  be  called  "cost 
growth".  Contract  modifications,  which  are  simply  changes  to  the  original  specifications 
delineated  in  the  contractual  agreement,  are  a  major  reason  for  cost  growth  in  a  contract. 
Cost  overruns  on  the  other  hand,  denotes  the  difference  between  actual  cost  experienced 
and  the  estimated  cost  delineated  in  the  contract;  i.e.  the  estimated  costs  remain  unchanged. 
[  Ref  4 ] 


2  As  will  be  seen  in  Chapter  Three,  the  development  costs  associated  with  the  CSTV 
project  also  includes  the  cost  to  produce  the  single  operational  test  vehicle. 
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Although  the  underling  purpose  of  this  paper  is  not  to  explore  the  various  cate¬ 
gories  of  contractual  agreements  that  exist  between  government  and  private  industry,  it  is 
useful  for  the  reader  to  acquire  a  broad  understanding  of  the  major  types  of  contracts  that 
are  utilized  in  support  of  state-of-the-art  extension  projects.  In  general,  a  contract  can  be 
seen  as  an  offer  and  acceptance  backed  by  legal  considerations.  The  types  of  contracts 
normally  used  to  support  RDT&E  efforts  (and  thus  include  state-of-the-art  extension  pro¬ 
jects)  are  discussed  below. 

Cost  contract-  A  cost  contract  calls  for  the  government  to  pay  all  allowable  costs 
involved  in  executing  a  given  research  project.  The  contractor  receives  no  fee.  This  type  of 
contract  establishes  an  estimate  of  the  total  costs  as  defined  in  the  contract  for  purposes  of 
(1)  obligating  current  funds,  and  (2)  establishing  a  ceiling  beyond  which  the  contractor 
cannot  proceed  (except  at  his  own  risk)  without  prior  approval.  [  Ref.  5:p.  16-3  ] 

Cost-Sharing  contract-  Under  a  cost-sharing  contract  the  contractor  is  reimbursed 
for  an  agreed  portion  of  his  allowable  costs,  not  to  exceed  an  established  ceiling  without 
fee.  [  Ref.  5:  p.  16-3  ] 

Cost-plus-fixed-fee  contract-  The  cost  plus  a  fixed  fee  contract  is  similar  to  the 
cost  contract  in  that  it  provides  for  payment  to  the  contractor  of  all  allowable  costs  as  de¬ 
fined  in  the  contract,  and  establishes  an  estimate  of  the  total  cost.  In  addition,  it  provides 
for  the  payment  of  a  fixed  fee  based  primarily  on  the  nature  of  work  to  be  performed  [  Ref 
5:p.  16-3  ].  As  will  be  seen  in  subsequent  chapters,  the  CSTV  project  involved  this  specific 
type  of  contractual  agreement. 

Cost-plus-incentive-fee  contract-  The  cost  plus  an  incentive  fee  contract  is  a  cost 
reimbursement  type  agreement  with  provision  for  a  fee  which  is  adjustable  by  formula  in 
accordance  with  the  relationship  which  total  allowable  costs  bear  to  target  costs.  Under  this 


type  of  contract,  there  is  negotiated  initially  a  target  cost,  a  target  fee,  a  minimum  and 
maximum  fee,  and  a  fee  adjustment  formula.  Factors  other  than  cost,  including  perfor¬ 
mance  and  progress,  can  also  be  used  as  a  basis  for  contract  incentive.  [  Ref.  5:p.  16-3  ] 


Fixed-price-incentive  contract-  The  fixed  price  incentive  contract  includes  a  pro¬ 
vision  for  the  adjustment  of  profit  and  the  establishment  of  the  final  contract  price  by  a  for¬ 
mula  based  on  the  relationship  which  final  negotiated  total  cost  bears  to  target  costs.  Under 
this  type  of  agreement,  target  cost,  profit,  price  ceiling,  and  a  formula  for  establishing  final 
profit  and  price  are  negotiated  prior  to  execution.  [  Ref.  5:p.  16-3  ] 

Firm-fixed-price  contract-  The  firm  fixed  price  contract  provides  for  a  price 
which  is  not  subject  to  any  adjustment  by  reason  of  the  cost  experience  of  the  contractor  in 
performance  of  the  contract.  This  type  of  agreement,  when  appropriately  applied,  places 
maximum  risk  upon  the  contractor.  Because  the  contractor  assumes  full  responsibility,  in 
the  form  of  profit  or  loss  for  all  costs  under  or  over  the  firm  fixed  price,  he  has  a  maximum 
profit  incentive  for  effective  cost  control  and  contract  performance.  The  firm  fixed  price 
contract  is  suitable  for  use  in  procurements  in  which  reasonably  definitive  design  and/or 
performance  specifications  are  known  and  fair  and  reasonable  prices  can  be  established  at 
the  outset.  This  type  of  contract  is  also  suitable  for  level-of-effort  work  in  which  the  con¬ 
tractor  is  compensated  for  expending  his  best  effort  at  fulfilling  program  requirements. 

[Ref.  5:p.  16-3  ] 

5.  Contract  Data  Requirements  List  (DP  Form  14231 

A  presentation  of  the  Contract  Data  Requirements  List  (CDRL)  is  appropriate 
here  since  it  will  be  referred  to  extensively  in  subsequent  chapters.  In  accordance  with  DOD 
instruction  5010.12,  all  government  contracts  which  require  data  as  a  deliverable  item  must 
have  these  requirements  delineated  in  a  Contract  Data  Requirements  List  (CDRL).  This 
documentation  must  state  all  data  requirements  that  the  contractor  is  to  furnish.  Some  of  the 


data  that  is  included  arc:  technical  status  reports,  cost  and  schedule  reports,  test  results, 
technical  manuals,  design  drafts  and  specification  listings,  and  specific  analyses.  For  many 
of  these  items,  an  approved  Department  of  Defense  identification  number,  known  as  a  Data 
Item  Description  (DID),  exists  and  must  be  cited.  [  Ref.  6:p.  4-52  ] 

C.  NAVY  COST  ESTIMATION  METHODS 

Inherent  in  all  Department  of  Defense  management  decisions  is  the  obligation  to  pro¬ 
vide  for  the  highest  mission  capability  possible  within  the  limits  of  available  resources.  The 
policy  on  overall  resource  allocation  is  stated  in  Department  of  Defense  directive  5000.1: 

A  cost  effective  balance  must  be  achieved  among  acquisition  costs,  ownership  costs . 

and  system  effectiveness  in  terms  of  mission  to  be  performed.  [  Ref.  7:p.  1  ] 

Throughout  the  Department  of  the  Navy,  there  is  much  importance  placed  on  accu¬ 
rately  estimating  the  probable  development  costs  associated  with  all  program  acquisitions, 
including  state-of-the-art  extension  projects.  Cost  estimating  efforts  are  found  in  every 
phase  of  the  Navy  planning,  programming,  and  budgeting  cycle  (PPBS)  as  well  as  phases 
within  the  acquisition  process.  However,  emphasis  on  cost  estimating  efforts  is  particularly 
predominant  in  the  planning  phase  of  the  PPBS.  The  accurate  estimation  of  development 
cost  is  an  essential  prerequisite  to  realistic  budgeting  for  weapon  systems.!  Ref  8:p.3  ] 

The  development  of  cost  estimates  for  a  particular  program  is  the  responsibility  of  the 
Principal  Developing  Activity  (PDA).  At  the  same  time,  independent  cost  estimates  are 
produced  by  the  Director  of  Navy  Program  Planning  (OP-90).  Finally,  the  DOD  Cost 
Analysis  Improvement  Group  (CAIG)  independently  conducts  a  comprehensive  review 
and  evaluation  of  both  estimates  and  provides  this  information  to  the  Joint  Requirements 
and  Management  Board  (JRMB).  The  primary  costing  methodologies  that  are  employed  by 
these  activities  are  ( 1 )  to  work  from  detailed  estimates  of  the  cost  of  work  packages  to 


derive  the  overall  estimate,  and/or  (2)  to  start  from  the  overall  characteristics  of  the 
particular  system  and  estimate  the  probable  development  costs  by  deductive  reasoning. 

[  Ref.  8:p.  10  ] 

The  detailed  estimation  approach  is  referred  to  as  the  Engineering  or  "bottom  up" 
method.  It  involves  breaking  down  the  project  into  separate  and  identifiable  segments  of 
work.  The  breakdown  is  accomplished  by  means  of  a  Work  Breakdown  Structure  (WBS). 
The  WBS  is  defined  in  Military  Standard  88 1 A  as: 

...  a  produce-oriented  family  tree  composed  of  hardware,  services  and  data  which  results 
from  project  engineering  efforts  during  the  development  and  production  of  a  defense  ma¬ 
teriel  item,  and  which  completely  defines  the  project/performance.  A  WBS  displays  and 
defines  the  product(s)  to  be  developed  or  produced  and  relates  the  elements  of  work  to  be 
accomplished  to  each  other  and  to  the  end  product.  (  Ref.  9  ] 

With  the  task  elements  of  the  project  identified  in  terms  of  the  work  breakdown  struc¬ 
ture,  development  costs  are  estimated  using  available  historical  cost  data  and  totalled  at  each 
level.  An  overall  developmental  cost  estimate  consists  of  a  summation  of  the  individual  de¬ 
velopment  costs  of  each  task  element.  [  Ref.  8:p.  8  ] 

The  second  major  cost  estimation  technique  used  by  the  Department  of  the  Navy  con¬ 
sists  of  initially  viewing  the  project  at  the  macro  level.  Specific  physical  and/or  performance 
characteristics  sought;  such  as  size,  complexity,  or  performance  level,  are  then  identified. 
Finally,  derived  relationships  known  as  Cost  Estimating  Relationships  (CER's)  are  applied 
to  the  parameters  of  the  project  to  develop  a  total  development  cost  estimate.  This  method  is 
known  as  Parametric  or  "Top  Down"  modeling.  The  method  (or  combination  of  methods) 
used  by  the  Navy  depends  on  various  factors  including  the  availability  of  relevant  historical 
data  and/or  the  complexity  of  the  project  being  considered.  [  Ref.  8:p.9  ] 
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D.  INDUSTRY-WIDE  COST  ESTIMATION  METHODS 


In  subsequent  chapters,  reference  is  made  to  cost  estimation  techniques  and  processes 
currendy  employed  by  Department  of  Defense  contractors.  The  parametric  and  engineering 
models  were  discussed  in  the  preceding  section.  These  two  models  are  extensively  em¬ 
ployed  within  private  industry  as  well.  What  follows  is  a  discussion  of  two  additional 
models  that  are  used  to  estimate  the  development  costs  associated  with  state-of-the-art  ex¬ 
tension  projects. 

1.  Associated  Program 

The  associated  program  method  of  cost  estimation  attempts  to  compare  the  total 
development  costs  of  a  current  project  with  that  of  a  technically  representative  program 
previously  completed.  If  the  two  projects  are  not  completely  identical,  an  attempt  is  made  to 
adjust  the  cost  estimate  to  reflect  similarities  and  differences  between  the  two  projects.  Be¬ 
cause  of  the  difficulty  in  obtaining  a  previous  project  that  can  satisfactorily  represent  the 
current  project,  the  associated  program  method  is  considered  the  least  accurate  method  of 
estimation.  It  is  most  often  used  to  obtain  initial  "staning  points"  early  in  the  conceptual 
phase  of  the  acquisition  process  and  when  little  or  no  historical  cost  data  over  a  wide  range 
of  programs  is  readily  available.  [  Ref.  10:p.  2-70  ] 

2.  Slmilar-To  Method 

The  similar-to  or  analog  method  of  cost  estimation  closely  resembles  the  associ¬ 
ated  program  method  but  with  an  important  difference.  Rather  than  basing  the  estimate  on 
the  cost  experience  of  a  single  representative  program,  a  historical  database  consisting  of 
actual  tasks  and  their  associated  costs,  is  utilized  to  estimate  the  development  cost  of  the 
current  project  [  Ref.  10:p.  2-70  ].  The  following  example  helps  clarify  the  point. 

The  use  of  new  structural  material  for  aircraft  often  requires  the  development  of  special 
cutting  and  forming  techniques  with  manufacturing  labor  requirements  that  differ  signifi¬ 
cantly  from  those  based  on  sample  primarily  aluminum  airframes.  Faced  with  this  prob¬ 
lem  when  titanium  was  first  considered  for  use  in  airframe  manufacture,  airframe 


companies  developed  standard-hour  values  for  titanium  fabrication  on  the  basis  of  shop 
experience  in  fabricating  test  pans  and  sections.  [  Ref.  1 1  ] 

The  cost  estimate  for  fabricating  titanium  airframes  was  based  on  an  analogous 
task  previously  experienced  when  titanium  test  pans  and  sections  were  fabricated.  The 
analogous  experience  does  not  come  from  a  single  past  project;  as  task  experience  gained 
from  numerous  projects  constitute  the  historical  cost  data.  In  fact,  the  similarity  of  the  end 
products  is  not  always  overly  relevant;  what  is  imponant  are  the  similar  tasks  themselves. 

Table  I  compares  the  major  cost  estimation  methods  currently  in  use  by  the  De- 
panment  of  the  Navy  and  defense  industry  contractors.  An  imponant  point  to  observe  is 
that  each  of  these  me  ,ods  have  inherent  advantages  and  disadvantages  and  are  used  either 
singularly  or  in  combination  depending  on  various  considerations.  Table  II  illustrates  the 
results  of  the  Booze,  Allen  and  Hamilton  study  of  prevalent  cost  estimation  techniques  as 
related  to  various  phases  of  the  acquisition  process  [  Ref.  12  ].  As  can  be  seen,  there  is 
evidence  to  suggest  that  the  cost  estimating  methods  employed  throughout  industry  are  not 
restricted  to  a  single  method, but  rather,  are  used  in  concert  with  one  another  to  generate  a 
development  and  production  cost  estimate.  This  concept  will  be  seen  in  a  later  chapter  when 
the  Lockheed  Missiles  and  Space  Estimating  System  Description  (ESD)  is  presented. 

E.  CHAPTER  SUMMARY 

This  chapter  outlined  several  concepts  considered  relevant  in  estimating  the  cost  of 
state-of-the-art  extension  projects.  A  workable  definition  of  state-of-the-art  extension  was 
established  followed  by  a  brief  look  at  development  programs  and  development  costs.  The 
imponant  difference  between  cost  growth  and  cost  overruns  (variances)  was  then  pre¬ 
sented.  Various  categories  of  government  contracts  used  in  the  Research,  Test,  Develop¬ 
ment.  and  Evaluation  (RTD&E)  effort  was  generally  reviewed.  Finally,  a  description  of  the 
most  prevalent  cost  estimating  methods  utilized  by  the  Department  of  the  Navy  and  the  De¬ 
fense  Industry  was  presented. 


TABLE  I 

COMPARISON  OF  ESTIMATION  METHODS  * 


Methodology 

Characteristics 

Pluses 

Minuses 

ASSOCIATED 

Based  on  total  actuals 

+  Provides  early 

-  Not  sensitive  to 

PROGRAM 

from  technically 
representative 

quick-look 

technical/program¬ 
matic  details  and 

program  modified  by 
economic  &  complexity 

+  Traceable  to 
histoncal 

differences 

factors 

costs 

-  Can't  provide  cost 
element  breakdown 

PARAMETRIC 

Derived  from  relation- 

+  Can  be  applied 

-  Not  traceable  to 

MODELING 

ships  of  cost  to  non 
cost  physical  or 

when  system 
characteristics 

actual  history 

performance 

are  known 

+  Can  identify 
cost  drivers 

-  Can't  easily 
validate 
algorithm 

SIMILAR-TO 

Generated  from 

High  degree  of 

-  Requires  relatively 

MODELING 

historical  cost  data 
using  tasks  or 

accuracy 

mature  design 

equipment  similar 

+  Provides  cost 

-  Requires  main- 

to  proposed/concept 

breakdown 

enance  of 

design 

in  WBS  format 

-1-  Models  use 
own  contractors 
data  base 

good  historical 
data  base 

-  Requires  cost 
analysts  with 
extensive  company 
experience 

ENGINEERING 

Detailed  functional  & 

+  High  degree  of 

-  Requires  detailed 

or 

"BOTTOM  UP " 
MODELING 

cost  element  estimates 
prepared  at  lowest 
practical  level  of  task 

accuracy 

design  description 

and  design  definitions 

-t-  Provides  WBS 
element  costs 
to  lowest  level 

-  Time  consuming 
and  expensive  to 
generate 

+  Uses  available  -  Can't  use  as  stand 
historical  cost  alone  approach 

data 


Source:  Information  received  during  interview  with  LMSC  Estimating  Systems  Manager,  Mr.  T.  Castro  on  24 
September  1987. 
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TABLE  II 

INDUSTRY  COST  ESTIMATING  METHODS  AND  MOST  PREVALENT 
USE  BY  ACQUISITION  PHASE 


EARLY 

FSD 


LATE 

FSD 


PRODUCTION 


PARAMETRIC 


ANALOGY 


ENGINEERING  N/A 


P=  Primary  Method(s)  CE=  Concept  &  Exploration  Phase 
S=  Secondary  Method(s)  D/E=  Demonstration  &  Validation  Phase 
N/A=  Not  Typically  Used  FS  D=Full  Scale  Development  Phase 
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III.  HISTORICAL  PERSPECTIVE,  PROGRAM  STRUCTURE 
AND  COST  ESTIMATION  PROCESSES 

A.  INTRODUCTION 

This  chapter  fc»cuses  on  the  specific  state-of-the-art  extension  project  chosen  for  the 
study,  i.e.  the  Control  System  Test  Vehicle  (CSTV)  developed  by  Lockheed  Missile  and 
Space  Company.  It  begins  by  describing  what  the  Control  System  Test  Vehicle  is,  what  it 
was  designed  to  accomplish,  and  what  Lockheed  Missiles  and  Space  Company's  contrac¬ 
tual  responsibilities  are.The  following  section  of  the  chapter  describes  the  overall  Lockheed 
Missiles  and  Space  Company's  (LMSC)  management  structure,  lines  of  communication 
and  authority,  and  the  specific  project  structure  incorporated  during  the  completion  of  the 
Control  System  Test  Vehicle  (CSTV)  project.  The  final  section  of  the  chapter  examines  the 
processes  and  methods  utilized  by  Lockheed  Missiles  and  Space  Company  in  estimating  the 
development  costs  associated  with  the  state-of-the-art  extension  project. 

B.  SYSTEM  DESCRIPTION 

Prior  to  the  age  of  nuclear  submarines,  the  general  understanding  of  submarine 
hydrodynamics  was  such  that  even  though  maneuvering  responses  were  not  always  as  ex¬ 
pected,  the  inherent  dangers  were  few  because  of  the  slower  operating  speeds  then  experi¬ 
enced.  However,  with  fleet  introduction  of  high  performance  nuclear  powered  attack  sub¬ 
marines,  in  particular  the  USS  Los  Angeles  (SSN-688),  these  operating  speeds  were 
greatly  increased  as  well  as  the  achievement  of  maneuvering  techniques  previously 
unattainable.  Extrapolation  of  data  from  computer  simulation  and  small-scale  models  to  de¬ 
termine  the  effect  of  submarine  design  modifications  on  the  performance,  stability  and  con¬ 
trol  were  found  to  yield  less  than  reliable  results.  The  ability  to  safely  and  fully  exploit  the 
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high  performance  potential  of  modem  high  speed  submarines  required  a  full  understanding 
of  all  factors  influencing  their  maneuverability  and  ability  to  recover  from  casualty  situa¬ 
tions.  To  this  end  the  Navy  desired  an  improved  method  whereby  data  pertaining  to  the 
control,  stability,  and  maneuverability  characteristics  of  the  Los  Angeles  class  attack  sub¬ 
marine  could  be  gathered,  analyzed,  and  understood. 

On  July  25,  1978,  Lockheed  Missiles  and  Space  Company  formally  responded  to  the 
Navy's  request  for  proposal  (RFP  number  N00024-78-R-5352S)  by  submitting  a  techni¬ 
cal,  management,  and  cost  proposal  calling  for  the  design,  construction,  and  test  of  a  con¬ 
trol  system  test  vehicle  and  related  support  equipment.  What  Lockheed  Missiles  and  Space 
Company  proposed  with  the  introduction  of  the  Control  System  Test  Vehicle  (CSTV)  was 
the  ability  to  provide  a  more  efficient  and  effective  means  of  assessing  maneuvering  re¬ 
sponses  to  control  inputs  with  an  accuracy  never  before  attained.  The  Lockheed  Missiles 
and  Space  Company  concept  involved  the  development  of  a  1/12  scale  model  of  the  Los 
Angeles  (SSN  688)  class  nuclear  submarine.  It  also  included  the  accessories  and  support 
equipment  needed  for  its  operation  and  maintenance.  Government  furnished  control  and 
instrumentation  components  were  to  be  contained  within  a  pressure  hull  and  fully  integrated 
with  all  other  vehicle  systems  to  form  an  unmanned,  free  running,  self-propelled  and  con¬ 
trolled  vehicle.  The  Control  System  Test  Vehicle  (CSTV)  was  designed  to  operate  in  fresh 
or  salt  water  to  depths  of  300  to  1,200  feet.  The  project  was  considered  by  Lockheed  Mis¬ 
siles  and  Space  Company  (LMSC)  to  be  a  technically  risky  endeavor.  This  was  primarily 
due  to  the  fact  that  much  of  the  technology  needed  to  successfully  complete  the  project 
would  have  to  be  integrated  in  a  way  never  before  attempted.  In  addition,  the  integrated 
electronics  and  government  furnished  computer  hardware  would  have  to  be  fitted  into  the 
confined  space  of  the  pressure  hull.  Funhenmore,  advancements  in  technologies  of  the  time 
were  required  in  the  areas  of  advanced  hydrodynamics,  propellers,  structures,  fabricating 


materials,  anticorosion  and  fouling,  and  submerged  mechanical  and  electronic  component 
integration. 

On  October  10,  1978,  LMSC  submitted  its  "Best  and  Final  Offer"  proposal  to  the 
Navy  program  manager  (Naval  Sea  Systems  Command)  with  a  cost  plus  fixed  fee  contract 
(  CSTV  contract  N00024-79-C-5356)  valued  at  $1,799,385  being  awarded  to  LMSC  on 
December  29,  1978. 

C.  CONTRACTOR  PROJECT  RESPONSIBILITIES 

The  responsibilities  and  obligations  of  LMSC  in  regards  to  the  CSTV  project  can  be 
found  by  referring  to  the  Statement  of  Work  (SOW).  The  complete  SOW  is  included  as 
Appendix  A.  In  broad  terms,  Lockheed  Missiles  and  Space  Company,  Inc.,  was  the  prime 
contractor  for  the  project  and  was  directly  responsible  for  accomplishing  the  following 
tasks: 

1 .  The  design  and  construction  of  one  (1)  Control  System  Test  Vehicle  (CSTV) 

2.  The  design  and  construction  of  a  set  of  suppon  equipment 

3 .  The  test  and  evaluation  of  the  vehicle  to  ensure  proper  operation 

4.  To  perform  system  engineering  activities  aimed  at  coordinating  and  controlling  de¬ 
velopment  and  ultimate  project  completion 

D.  ORGANIZATIONAL  STRUCTURE 

For  the  Control  System  Test  Vehicle  Project.  Lockheed  Missiles  and  Space  Company 
(LMSC)  drew  upon  the  management  success  previously  realized  by  the  Lockheed  Califor¬ 
nia  Company  located  in  Burbank,  California  (a  wholly  independent  subsidiary  of  the  parent 
company.  The  Lockheed  Corporation)  and  it's  highly  successful  "Skunk  Works"  project 
team  approach.  I*rojcct  organizations  structured  along  task-team  lines  are  utilized  by  all 
Lockheed  subsidiaries  for  many  of  their  advanced  development  or  state-of-the-art  extension 
programs.  The  CSTV  project  was  therefore  a  fitting  candidate  for  this  project  team  ap¬ 
proach.  The  decision  to  organize  the  project  in  this  manner  was  based  on  an  internal 


assessment  of  management  elements  seen  as  vital  to  the  success  of  the  CSTV  project. 
These  management  elements  are  presented  as  follows: 


TABLE  II 

MANAGEMENT  ELEMENTS  SEEN  AS  VITAL  TO  THE  CSTV  PROJECT 


*  Authorized  Project  Manager,  reporting  to  a  Company  Officer,  full  authority  for 
his  Project 

*  Establish  a  small  and  highly  competent  project  organization 

*  Use  a  flexible  and  low  cost  drawing  system  that  expedites  and  controls  changes 

*  Keep  documentation  to  a  minimum,  meeting  CDRL  requirements 

*  Maintain  continuous  status  of  the  project  visible  to  management 

*  Establish  an  austere  and  fully  compliant  Quality  Assurance  Program 

*  Support  and  participate  in  post-delivery  testing  to  maintain  competence  for 
follow-on  support 

*  Earn  customer  trust  with  integrity  and  close  cooperation  and  liaison 

*  Obtain  customer  understanding  and  concurrence  at  the  stan  of  the  contract  on 
methods  and  procedures  for  controlling  government  furnished  equipment  and 
echnical  data 

*  Limit  project  access  and  the  number  of  support  personnel  required 


The  utilization  of  a  project  team  approach  was  beneficial  in  controlling  previous  SOA 
project  costs.  Lockheed  Corporation  believes  that  the  small  number  of  people  within  a  pro¬ 
ject  team  significantly  improves  management  coordination  and  control.  Furtheimore,  cost 
control  responsibilities  within  the  project  manager's  organization  greatly  increases  cost 
awareness  and  the  desire  for  controlling  costs. 


The  overall  organizational  structure  of  the  Lockheed  Corporation  is  depicted  in 
Figure  1,  which  also  shows  the  relationship  between  the  Control  S>  stem  Test  Vehicle  pro¬ 
ject  team  and  the  other  organizational  elements  of  Lockheed.  The  Lockheed  Missiles  and 
Space  Company,  Inc.  (LMSC)  is  headquanered  in  Sunnyvale,  California.  Within  LMSC, 
the  Research  and  Development  Division  (R&DD)  is  responsible  for  developing  new  and 
advanced  products  that  take  maximum  advantage  of  the  skills  available  within  the  corpora¬ 
tion.  Ocean  Systems  is  one  of  the  major  product  lines  in  R&DD.  The  CSTV  project  man¬ 
agement  team  was  placed  directly  within  Ocean  Systems.  Within  this  structural  arrange¬ 
ment,  the  CSTV  project  team  was  authorized,  and  in  fact  encouraged,  to  draw  upon  all 
three  major  functional  organizations  within  LMSC  (  Missile  Systems,  Research  and  Devel¬ 
opment,  and  Space  Systems)  for  technical  development  support  and  expertise. 


As  was  previously  stated,  Figure  1  illustrates  the  placement  of  the  CSTV  project 
within  the  overall  Lockheed  organization.  This  arrangement  provided  direct  access  to  top 
executives  within  the  corporation.  This  directness  in  communications  was  confirmed  in 
several  personal  interviews  with  LMSC  managers  involved  with  the  CSTV  project  and  was 
considered  an  important  advantage  during  the  entire  project.  Also  of  significance  was  the 
authority  given  to  the  project  management  team  during  development  of  the  CSTV  project. 
Again,  this  seemed  to  be  a  positive  aspect  of  the  management  structure,  as  expressed  by  the 
individuals  associated  with  the  project. 


The  overriding  concept  of  the  CSTV  project  management  effort,  i.e.  tight  project 
controls,  and  full  authority  and  responsibility  of  the  project  manager,  is  reflected  in  the 
project  team  organizational  chart  (Figure  2).  The  project  team  structure  can  be  seen  as  a 
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Figure  1.  Placement  of  the  CSTV  Project  Team  within  Lockheed 
Source.  LMSC  Management  proposal  no.  D085091  of  25  July  1978 
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Reports  to  J.G.  Wenzel ,  Vice  President,  Ocean  Systems 


(1410,  1440,  1450,  (1300  ALL) 

1460,  1470,  1530) 

( )  =  WBS  number 


Figure  2.  The  CSTV  Project  structure  end  WBS  task  responsibilities 

Source:  LMSC  Management  proposal  no.  D085091  of  25  July  1978 
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hardware-oriented  organization  keyed  to  the  Work  Breakdown  Structure  (WBS).  Major 
tasks  and  hardware  elements  are  grouped  based  on  similarity  of  content.  Composed  of  key 
Ocean  Systems  engineers  and  Naval  Architects,  the  Senior  Technical  Advisory  Board, 
provided  an  important  service  for  the  CSTV  project  team.  The  board  met  quarterly  and  at 
the  request  of  the  CSTV  project  manager,  to  provide  an  independent  audit  and  evaluation  of 
technical  and  management  progress.  REMCO  Hydraulics  of  Willets, California,  the  prime 
subcontractor  was  officially  considered  part  of  the  CSTV  project  team.  As  developer  of  the 
CSTV's  pressure  hull,  it  was  felt  that  close  coordination  and  control  of  this  most  critical 
subassembly  would  best  be  achieved  in  this  manner. 

E.  WORK  BREAKDOWN  STRUCTURE 

A  CSTV  Work  Breakdown  Structure  was  used  to  control  task  definitions,  work  as¬ 
signments,  budget  allocations,  development  costs,  and  the  assignment  of  personnel  re¬ 
sponsibilities.  It  was  seen  as  the  master  guide  for  identifying  and  evaluating  all  activities 
within  the  CSTV  project.  What  follows  is  a  description  of  the  major  elements  of  the  WBS 
along  with  corresponding  task  identification  numbers. 

1100-  MODEL  ( the  Control  System  Test  Vehicle  itself).  This  element  included  all  the 
technical  analysis,  design,  fabrication,  and  procurement  effort  required  to  design,  build, 
and  deliver  the  complete  CSTV  vehicle  incorporating  both  contractor  furnished  equipment 
(CFE)  and  government  furnished  equipment  (GFE)  items.  Also  included  was  the  effon  re¬ 
quired  to  prepare  and  deliver  the  related  disclosure  information,  drawings,  and  reports 
called  for  by  the  Contract  Data  Requirements  List  (CDRL).  Subelements  included:  Hull 
and  Appendages;  Propulsion  Subsystems;  Power  System;  Control  and  Recording  System; 
Navigation  System;  Auxiliary  Systems;  and  Project  Integration  and  Assembly.  The  work 
on  the  last  task  element  was  completed  when  the  CSTV  vehicle  was  ready  for  integrated 
systems  tests. 
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1200-  SUPPORT  EQUIPMENT.  This  element  included  all  technical  analysis,  design, 
fabrication,  and  procurement  efforts  required  to  design,  build,  and  deliver  the  suppon 
equipment  needed  for  the  full  operation  of  the  CSTV  System.  The  effon  also  included  all 
documentation,  reports,  interface  coordination,  and  liaison  with  the  government  and  GFE 
suppliers.  Subelements  included:  Control  and  Display  System;  Trailer,  Sled;  Dollies;  Ship¬ 
ping  Containers;  Battery  Chargers;  Auxiliary  Power  System;  and  Miscellaneous  Equip¬ 
ment. 

1300-  TEST  AND  EVALUATION.  This  element  covered  all  efforts  involved  in  plan¬ 
ning  and  conducting  required  operational  tests,  including  preparation  of  plans,  actually 
conducting  the  test,  the  analysis  of  the  results,  and  the  preparation  of  the  test  reports  as  re¬ 
quired  by  the  CDRL.  Subelements  included:  Development,  System,  and  Acceptance  Tests. 

1400-  SYSTEM  ENGINEERING.  Efforts  under  this  element  included  those  resources 
required  to  analyze  and  define  the  systems  detailed  requirements  and  the  establishment  and 
control  of  interfaces.  It  included  defining  and  monitoring  the  integration  of  the  vehicle,  its 
support  equipment,  and  all  GFE  components.  Subelements  included:  Systems  Engineering; 
System  Safety;  Quality  Assurance,  Reliability,  and  Maintainability  Programs;  Human  En¬ 
gineering;  and  Mockups. 

1500-  SYSTEMS  MANAGEMENT.  This  element  covered  overall  FYoject  Manage¬ 
ment  activities  including  technical  and  management  direction,  project  controls,  status 
evaluation,  establishment  of  priorities,  assignment  of  tasks,  and  other  appropriate  project 
management  functions.  Subelements  included:  Project  Management;  Configuration  and 
Data  Management:  and  Integrated  Logistics  Support. 

Task  elements  of  the  WBS  are  assigned  a  unique  charge  number  identified  as  a  work 
order/work  authority  (WO/WA);  which  is  also  an  element  of  the  LMSC  cost  accounting 
system.  This  charge  number  corresponds  to  the  WBS  series  and  is  further  identified  within 
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the  accounting  system  in  terms  of  organization  number,  primary  cost  element,  and  resource 
code.  All  budgeting,  scheduling,  cost  collection,  and  measurement  of  work  status  begins  at 
the  cost  account  level.  Thus  the  WBS  is  also  the  primary  management  control  mechanism 
for  summarizing  various  project  information  for  reporting  or  monitoring  purposes. 

F.  LMSC  COST  ESTIMATION  SYSTEM 

What  follows  is  a  description  of  the  generic  process  used  by  LMSC  to  estimate  the  de¬ 
velopment  and  production  costs  of  all  projects,  including  those  associated  with  technologi¬ 
cal  extensions  of  the  state-of-the-art.  This  process  is  the  actual  series  of  steps  employed  by 
the  CSTV  project  team  in  estimating  the  project's  development  costs.The  specific  estima¬ 
tion  method(s)  used  in  the  CSTV  project  along  with  any  significant  deviations  from  the 
generic  LMSC  process  are  examined  in  the  following  section. 

The  LMSC  cost  estimation  process  can  be  seen  as  a  series  of  events,  within  phases, 
utilizing  the  various  cost  estimation  techniques  described  in  Chapter  II  in  an  effort  to  pro¬ 
duce  effective  and  creditable  cost  estimates.  The  primary  considerations  of  the  system  are 
to: 

*  Ensure  that  source  data  for  estimates  are  current,  accurate,  and  complete 

*  Develop  and  maintain  documentation  in  support  of  the  estimates 

*  Assign  responsibilities  for  originating,  reviewing,  and  approving  the  estimates 

*  Utilize  successful  techniques  for  developing  direct  and  indirect  cost  estimates 

1.  Pre-ProDosal  Planning 

Figure  3  illustrates  the  key  elements  involved  in  the  pre-proposal  planning  phase. 
The  cost  estimation  process  begins  when  a  Program  Manager  is  assigned  to  a  new  business 
activity.  In  the  initial  planning  phase  of  a  project,  the  Program  Manager  will  form  a  com¬ 
bined  Program  Office  Staff  and  a  contract  proposal  team.  This  combined  team  is  organized 
with  clearly  defined  lines  of  communications  and  authority.  The  members  of  the  proposal 
team  consist  of  representatives  from  various  functional  divisions  within  the  LMSC 
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organization  who  arc  believed  to  possess  the  specific  skills  and  experience  to  plan  the 
program  and  proposal  efforts  and  to  reliably  estimate  the  projects'  development  and 
production  costs.  A  teaming  agreement  ^  is  authorized  if :  1 )  the  capabilities  required  are 
not  available  within  LMSC,  or  2)  the  unique  capabilities  of  LMSC  personnel  coupled  with 
non-LMSC  team  members  are  highly  complementary  and  afford  the  customer  the  best 
combination  of  capabilities  required  to  achieve  system  pe.'formance  and  cost  objectives. 
The  CSTV  project  did  not  require  this  arrangement  as  all  capabilities  were  considered 
available  within  LMSC.  Requests  for  Proposals  (RFP)  are  used  to  communicate  the 
government's  requirements  for  a  new  contract.  Once  received,  the  proposal  team  begins  the 
next  phase  of  effort. 

2.  Proposal  Requirements  Review  and  Development 

This  phase  of  the  estimation  process  involves  a  comprehensive  review  and  inter¬ 
pretation  of  the  technical  and  management  requirements  delineated  in  the  RFP.  Figure  4  is 
provided  to  include  a  complete  presentation  of  the  numerous  events  associated  with  this 
phase  of  the  process.  Only  the  most  essential  concerns  will  be  discussed. 

The  first  action,  the  Proposal  Schedule,  established  the  ground  rules  and  future 
milestones  that  are  to  be  incorporated  in  the  overall  cost  proposal  development  and  plan  and 
schedule.  An  important  objective  of  the  proposal  team  is  for  cost  proposals  to  identify  the 
conditions  and  assumptions  taken  into  consideration  during  its  preparation.  This  is  the  fo¬ 
cus  of  the  RFP  review  process.  Its  intent  is  to  single  out  those  task  elements  or  conditions 
that  may  impact  highly  on  contract  costs.  The  specific  items  of  consideration  include 


-  A  teaming  agreement  is  a  legally  binding  written  agreement  with  another  company 
(outside  LMSC)  to  jointly  prepare  or  conduct  marketing  research  or  development  efforts 
leading  to  the  award  of  a  new  contract. 
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Figure  4.  Proposal  Requirements  Review  and  Development 
Phase 


Source:  LMSC  Estimating  System  Description  (ESD)  Manual 
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the:  1)  review  of  contract  deliverables,  2)  examination  of  contract  options,'*  3)  review  of 
the  terms  and  conditions,  4)  review  of  any  funding  constraints,  5)  review  of  the  GFE  list, 
6)  review  SOW,  WBS  and  WBS  dictionary  (  task  elements)  and  7)  review  of  CDRL 
requirements.  Depending  on  project  complexity,  the  degree  to  which  technological  methods 
will  be  required  to  be  advanced,  and  the  stringency  or  difficulty  of  the  RFP  requirements, 
additional  levels  of  program  scheduling  and  review  may  be  conducted  to  define  more 
detailed  activities.  If  required,  this  phase  would  occur  somewhat  after  (but  in  some  cases 
parallel  to)  the  review  phase.  As  part  of  the  program  scheduling  and  planning  task,  a  make- 
or-buy  analysis  is  also  conducted.  Often  this  is  a  formal  requirement  stated  in  the  RFP.  The 
CSTV  project  however,  did  not  require  this  documentation,  presumably  because  it  was 
considered  a  development  project  involving  no  foliow-on  production. 

One  of  the  most  important  LMSC  documents  resulting  from  this  phase  is  the 
Program  Requirements  List  (PRL).  This  list  contains  all  equipment,  subcontracts,  and 
software  needed  for  pricing.  Also  often  included  with  the  PRL  is  descriptive  data  on  all 
listed  items.  These  are  known  as  fact  sheets,  and  are  put  together  by  engineering  divisions 
and  are  used  in  "bottom-up"  cost  estimation.The  final  two  steps  within  this  phase  of  the 
estimation  process  involve  issuing  quoting  instructions  and  defining  pricing  logic  ground 
rules.  These  instructions  are  intended  to  produce  consistency  and  prevent  confusion  during 
the  next  phase  of  the  process;  the  proposal  estimation  process. 

3.  Proposal  Estimation 

This  section  describes  the  LMSC  process  for  determining  and  developing  valid 
cost  rationale  and  estimation.  Figures  5A  include  all  of  the  major  steps  involved  in  this 


“*  A  contract  option  is  a  unilateral  right  in  a  contract  by  which,  for  a  specific  time,  the 
government  may  elect  to  purchase  additional  supplies  and  services  called  for  in  the  contract, 
or  may  elect  to  extend  the  terms  of  the  contract. 
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phase.  For  the  purposes  of  this  paper,  only  those  steps  used  by  the  CSTV  project  team  and 
any  other  essential  steps  will  be  reviewed.  It  is  also  worth  mentioning  here  that  a  significant 
number  of  the  steps  involved  in  the  LMSC  estimation  process  are  designed  to  satisfy  the 
voluminous  documentation  requirements  established  by  the  Federal  Acquisition  Regula¬ 
tions  (FAR)  as  opposed  to  generating  pure  estimations  of  development  and  production  cost 
of  a  particular  project.  Documentation  requirements  of  the  Federal  Acquisition  Regulations 
are  beyond  the  scope  of  this  study.  Therefore,  the  process  involved  will  be  described  in 
terms  of  the  steps  whose  objectives  are  solely  that  of  estimating  development  and  produc¬ 
tion  costs. 

This  phase  in  the  process  begins  with  a  "kick-off  meeting  where  cost  input 
scheduling,  cost/price  estimating  procedures,  and  other  administrative  preliminaries  are  es¬ 
tablished  by  the  program  manager.  Once  these  ground  rules  are  established,  the  actual  cost 
estimation  work  begins. 

The  total  cost  estimate  is  composed  of  three  major  cost  categories.  These  are:  di¬ 
rect  materials  (and  its  corresponding  overhead  costs),  direct  labor  (and  its  corresponding 
overhead  costs),  and  other  direct  costs.  The  material  costs  are  further  broken  down  into 
three  categories  and  involve  the  following  actions; 

A.  Subcontracted  material  and  purchased  services:  This  category  includes  project  spe¬ 
cific  parts,  components,  reworked  items  and  test,  and  consulting  services  that 
are  not  to  be  manufactured  or  provided  directly  by  Lockheed  itself.  The  solicitation 
of  various  subcontractors  is  done  via  formal  Lockheed  RFP's  or  informal  request 
for  quotations  (RFQ).  Often  there  is  insufficient  time  to  obtain  subcontractor  cost 
data  before  the  Lockheed  proposal  due  date.  Moreover,  as  is  the  case  with  state-of- 
the-art  developments,  subcontractor  supplied  materials  and  the  corresponding  price 
quotations  are  not  available.  In  this  case,  Locklieed  will  prepare  an  in-house 


estimate  based  on  a  quote  for  similar  effon  from  another  subcontractor  (analogy  or 
similar-to  method),  or  purchase  order  history  from  a  similar  contract  (bottom-up 
method). 

B .  Standard  commercial  material  and  raw  material:  This  category  includes  standard 
items  that  Lockheed  normally  fabricates,  in  whole  or  in  part,  and  that  are  generally 
stocked  in  inventoiy.  Raw  material  consists  of  material  in  a  form  that  requires 
funher  processing.  Supplier  quotations  are  the  preferred  method  of  obtaining  cost 
data  however,  often  as  is  the  case  with  state-of-the-art  developments,  in  house 
estimation  is  again  utilized. 

C.  Intra-Lockheed  work  transfer  (IWr):  This  category  includes  all  material  items  that 
are  fabricated  by  Lockheed  itself.  The  raw  materials  for  these  items  are  carried  in 
Lockheed  inventory  and  hence  are  accounted  for  at  actual  costs.  No  material 
estimation  is  therefore  needed. 

An  important  step  prior  to  the  forwarding  of  a  final  consolidated  Bill  of  Material 
(BOM)  is  a  process  called  Price  Analysis.  This  auxiliary  analysis  is  performed  whenever 
(1)  the  total  BOM  amount  exceeds  $100,000  and/or  (2)  the  rate  of  technological  change  of 
the  project  is  believed  to  include  significant  uncertainty  as  to  justify  additional  review  and 
consideration  of  the  reasonableness  of  the  cost  estimation.  The  specific  actions  taken  in 

conducting  a  Price  Analysis,  including  the  following: 

*  Comparison  of  price  quotation  received 

*  Comparison  of  prior  price  quotations  with  current  current  quotations  for  the  same  or 
similar  end  items 

*  Employing  parametric  modeling  techniques  as  "sanity"  checks,  i.e.  do  these  prices 
appear  reasonable? 

*  Comparing  prices  set  forth  in  published  price  lists  with  discount  or  quantity  buy  op¬ 
portunities 

*  Comparison  of  proposed  prices  with  independently  developed  Lockheed  estimates 
(similar-to  method) 
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The  end  product  of  the  material  cost  estimation  process  described  above  is  a 
consolidated  Bill  of  Material  (BOM)  composed  of  the  individual  cost  categories.  Final  ac¬ 
tion  on  the  BOM  involves  the  application  of  a  material  escalation  rate  depending  on  the  an¬ 
ticipated  length  of  the  particular  project. 

The  next  category  of  cost  to  be  considered  is  that  of  direct  labor.  Figure  5B 
illustrates  the  steps  involved  in  this  phase  of  the  process. The  estimates  associated  with 
direct  labor  are  prepared  and  expressed  in  labor  hours  traceable  to  the  research,  design, 
development,  and  production  of  a  particular  project.  Labor  hour  requirements  for 
completion  of  the  work  to  be  performed  as  outlined  in  the  statement  of  work  (SOW)  or 
w'ork  breakdown  structure  (WBS)  are  estimated.  There  are  two  basic  methods  used  to 
arrive  at  these  labor  hour  estimates.They  are  :  (1)  the  Similar-To  or  Analog  method;  or  (2) 
the  Engineering  or  "Bottom-Up"  method.  The  Similar  -To  method  is  the  preferred 
technique.  It  generally  yields  the  most  accurate  estimate  if  properly  executed.  However, 
this  method  requires  the  availability  of  extensive  historical  data  upon  which  to  base  the  es¬ 
timate.  The  CSTV  project  along  with  many  state-of-the-art  extension  projects  may  involve  a 
significant  number  of  tasks  in  which  little  or  no  similar  data  exists.  Therefore,  the 
Engineering  method  is  generally  used  to  estimate  the  labor  costs  associated  with  state-of- 
the-art  extension  projects.  The  CSTV  project  involved  the  utilization  of  both  methods, 
however  much  of  the  labor  estimation  was  based  upon  engineering  estimates.  Once  direct 
labor  hours  are  estimated,  they  are  converted  to  dollar  costs  by  means  of  applying  ( I )  cost 
standards;  and  (if  appropriate)  (2)  learning  curve  theory.  The  cost  standards  are  based  on 
relating  development  and  production  costs  to  the  specific  characteristics  of  the  project  (such 
as  composition,  weight,  size,  or  duration)  and  applying  Lockheed  and/or  industry-wide 
statistics  as  appropriate.  In  general,  learning  curve  theory  states  that  the  amount  of  time 
required  to  complete  a  given  task  will  diminish  over  time  as  workers  gain  added  experience 
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performing  a  given  task.  LMSC  obtains  the  specific  rate  of  learning  for  each  labor  task  by 
employing  the  least-squares  method  of  curve  fitting  to  a  LMSC  specific  labor  cost 
database.  Although  learning  curves  may  be  correctly  applied  to  some  projects  involving 
state-of-the-an  extensions,  the  CSTV  project  did  not  include  the  application  of  a  learning 
curve  rate.  This  was  apparently  due  to  the  unique  nature  of  the  project  and  the  fact  that  no 
follow-on  production  was  required. 

The  final  category  of  cost  estimated  is  that  associated  with  travel,  ovenime 
premiums,  and  other  direct  costs  not  previously  included.  Typically  these  costs  are  arrived 
at  by  similar-to  techniques  or  direct  quotations  from  supplying  sources.  The  final  phase  of 
the  LMSC  cost  estimating  system  involves  a  process  of  extensive  management  reviewout- 
side  the  auspices  of  the  project  team.  Post-estimation  phase  techniques  and  considerations 
are  the  subject  of  the  following  section. 

G.  OTHER  CONSIDERATIONS  AND  COST  MODELING 

A  series  of  management  reviews  are  conducted  prior  to  the  final  proposal  and  follow¬ 
ing  the  approved  cost  estimates  developed  during  the  proposal  estimating  process.  LMSC 
utilizes  a  parametric  cost  model;  the  RCA  PRICE  (Hardware)  model,  and  a  combined  anal¬ 
ogy/associated  model;  the  Lockheed  STAR  model.  Discussion  in  this  section  is  not  meant 
to  imply  that  their  use  is  reserved  solely  for  the  later  phases  of  the  estimation  process. 
These  methods  are  used  to  provide  credibility  or  "sanity"  checks  throughout  the  entire  ac¬ 
quisition  process.  This  concept  is  illustrated  in  Figure  6. 

As  can  be  seen,  the  "mix"  of  cost  estimation  techniques  utilized  by  LMSC  is  time  phased 
and  correlated  with  the  project  managers  view  of  cost  uncertainty.  What  follows  is  a  brief 
presentation  of  the  two  methods  used  by  LMSC  to  provide  for  creditability  checks 
throughout  the  cost  estimation  process. 


COST 


Figure  6:  The  use  of  various  cost  estimation  techniques 
throughout  the  acquisition  process 


Source:  Information  received  during  interview  with  LMSC  Etimating  Systems 

Manager.  Mr.  T.  Castro  on  24  September  1987. 
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The  parametric  model  known  as  the  RCA  PRICE  (Hardware)  model  is  a  com¬ 


mercially  available  decision  support  system  that  is  appropriate  for  estimating  development 


and/or  production  costs  at  the  subassembly  or  higher  level.  It  is  not  appropriate  for  use 


with  extremely  small  component  parts.  Input  into  the  model  is  achieved  by  means  of  mi¬ 


crocomputer  with  direct  access  through  data  link  (MODEM)  to  the  RCA  PRICE  mainframe 


computer.  Input  parameters  include: 

( 1 )  Physical  factors;  component  weight  in  two  domains;  (a)  the  active  electronic  weight 
and  (b)  the  mechanical/structural  weight.  These  two  input  variables  are  consider^ 
critical  to  the  accuracy  of  the  RCA  PRICE  output.  Additional  physical  factors 
include  the  volume  and/or  density  of  the  component.  These  two  inputs  are  not  as 
critical  to  output  accuracy. 


(2)  Qualitative  inputs:  component  empirical  data  in  three  variables;  (a)  the  electronic 
complexity,  (b)  the  scructural/mechanical  complexity,  and  (c)  the  engineering  com¬ 
plexity.  The  critical  variable  for  developmental  projects  is  the  engineering 
complexity  input;  which  pertains  to  the  scope  of  hardware  development  task  and  the 
skills  of  the  project  team.  Values  for  these  quantitative  factors  are  provided  in 
matrix  form  by  the  model. 

(3)  Other  inputs:  These  parameters  incorporate  idea  such  as  design  repetition,  number 
of  prototypes  to  be  developed,  end  units  to  be  produced,  production  learning 
curves,  and  economic  factors  for  escalation/de-escalation  of  labor  rates. 


Input  variables  are  completed  for  all  identifiable  hardware  items  of  the  system 


and  are  entered  into  the  RCA  PRICE  model.  The  cost  output  received  by  the  model  is  bro¬ 


ken  down  into  hardware  development,  production,  and  total  costs.  Input  parameters  are  re¬ 


peated  and  "what  if  analysis  is  easily  accomplished  and  thus  cost  categories  that  signifi¬ 


cantly  affect  total  costs,  i.e.  cost  drivers,  can  be  identified.  Several  limitations  of  the  RCA 


PRICE  model  are  apparent.  The  most  obvious  is  that  input  parameters  require  that  system 


characteristics  be  known  a  priori  to  the  estimate.  This  severely  limits  its  application  to 


state-of-the-art  extension  projects  in  that  many  of  the  characteristics  are  unknown  or  at  best 


subjective  estimates  themselves.  The  accuracy  of  the  output  is  also  highly  dependent  on  the 


input  parameters.  In  addition,  the  actual  algorithm  employed  is  considered  proprietary  in 
nature  and  thus  is  not  easily  validated.  One  would  expect  to  find  significant  skepticism 
associated  with  this  method  of  estimation.  However,  interviews  with  LMSC  cost  estima¬ 
tors  conducted  by  the  researcher  revealed  surprising  support  and  credibility  for  the  model. 

2.  Lockheed  STAR  Model 

The  Lockheed  STAR  cost  estimation  model  is  a  combination  of  the  characteris¬ 
tics  inherent  in  both  the  associated  and  the  similar-to  models.  The  primary  focus  in  the 
STAR  model  is  the  extensive  use  of  detailed  databases  drawn  from  a  large  number  of  past 
Lockheed  projects  over  a  twenty  year  period.  Three  separate  and  distinct  databases  are 
maintained  and  used  to  generate  cost  estimates.  Depending  on  the  project  under  considera¬ 
tion,  the  Flight  Hardware,  Ground  Hardware,  or  Software  databases  are  accessed  to  cost 
data  pertaining  to  a  particular  element  or  task  being  costed.  The  estimation  model  matches 
the  task  and  project  of  interest  to  functionally  analogous  programs  and  historical  cost  expe¬ 
riences  found  within  the  database.  Each  model  (Right  Hardware,  Ground  Hardware,  or 
Software)  uses  an  algorithm  tailored  to  its  own  specific  application  and  reflects  the  way  in 
which  Lockheed  does  business.  If  required,  all  three  application  programs  can  be  integrated 
to  produce  a  total  system  cost  estimate.  The  STAR  model  databases  do  not  ir.  :!ude  ocean 
systems  project  experience  and  hence  was  not  utilized  in  the  CSTV  project.  However,  it  is 
included  in  the  discussion  due  to  its  extensive  current  use  by  LMSC  cost  estimators. 

H.  SUMMARY  OF  THE  COST  ESTIMATION  PROCESS  USED  FOR  THE 

CSTV  PROJECT 

The  CSTV  project  closely  followed  the  LMSC  estimating  system  process  previously 
discussed.  What  is  now  discussed  is  the  estimating  methods  utilized  in  developing  the 
CSTV  project  cost  proposal.  The  predominant  method  used  by  LMSC  to  estimate  the  de¬ 
velopment  costs  for  the  CSTV  contract  proposals  was  the  engineering  or  bottom-up 
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technique.  Past  experience  with  costs  associated  with  similar  projects  was  relied  upon 
when  comparisons  were  considered  appropriate  ^ .  Often,  however,  the  advanced  technical 
nature  of  the  CSTV  project  prevented  direct  applications  and  various  adjustments  were 
made  in  an  attempt  to  improve  estimation  validity. 

LMSC's  cost  proposal  for  the  Cost-Plus  Fixed  Fee  (CPFF)  contract  used  in  support  of 
the  CSTV  acquisition  contained  the  following  explanations  of  the  development  costs  in¬ 
cluded  and  how  these  costs  were  derived: 

1.  Direct  Materials;  TTie  material  requirements  for  the  project  were  directly  estimated 
based  upon  the  Work  Breakdown  Structure  (WBS)  and  the  Statement  of  Work 
(SOW)  documents.  The  costs  were  based  on  vendor  quotes,  catalog  prices,  previous 
purchase  order  history,  and  bottom-up  engineering  estimates.  Common  minor  mate¬ 
rials  (standard  or  common  productive  parts  having  broad  applicability  and  wide  us¬ 
age)  were  applied  to  direct  materials  by  a  negotiated  fixed  government  rate. This  rate 
is  a  LMSC  standard  rate  based  on  historical  costs. 

2.  Material  Overhead:  This  cost  element  represented  the  administrative  burden  associ¬ 
ated  with  material  and  subcontract  costs  in  two  categories;  material  and  procurement. 
Prenegotiated  fixed  rates  based  on  historical  costs  were  applied  to  each  item. 

3.  Direct  Labor:  The  estimated  direct  labor  hour  requirements  for  the  CSTV  project  were 
derived  from  engineering  estimates  made  by  project  team  membcrs.Task  require¬ 
ments  were  identified  by  analysis  of  the  CSTV  project  WBS.  Labor  classifications 
were  made  as  follows:  Staff  Engineers,  Specialized  Engineers,  General  Engineers. 
Inspection,  Technical  Publications,  and  Manufacturing  Development.  Hourly  labor 
rates,  provided  by  LMSC's  company-wide  pooled  labor  rate  database  system,  are 
applied  to  each  labor  classification  total.  These  rates  were  incrementally  escalated  by 
cost  of  living/merit  increase  factors  expected  to  be  incurred. 

4.  Labor  Overhead:  Labor  overhead  costs  are  divided  into  two  categories;  development 
overhead  and  manufacturing  overhead.  Specific  rates  were  applied  to  the  labor  hour 
estimates  of  each  category.  These  rates  are  company-wide,  pre-negotiated  and  based 
upion  historical  LMSC  cost  experience. 

5.  Travel  Costs:  Direct  travel  costs  arc  based  on  the  number,  destination,  and  duration 
of  trips  expected  to  be  required  to  satisfy  the  requirements  for  testing  and  special 
material  item  procurement.  LMSC  standard  costs,  based  on  tourist  class,  round  trip 
airfares  and  government  approved  per  diem  rates,  were  used. 


5  These  similar  experiences  in  which  the  CSTV  project  team  referred  to  included  the 
Deep  Submergence  Rescue  Vehicle  (DSRV),  the  Summa  Ocean  Mining  Barge,  and  the 
Deep  Quest  Research  Submersible  projects. 


r-j, 


6.  Other  Direct  Costs:  This  category  included  four  classifications  to  include  reproduc¬ 

tion,  computer,  facility  capital  cost  of  money,  and  overtime  premium  costs.  Repro¬ 
duction  included  labor,  materials,  and  vendor  support  costs  associated  with  the  pro¬ 
cess  of  reproducing  technical  drawings,  project  plans,  and  blueprints.  The  rate  per 
hour  applied  to  tot^  labor  hours  was  a  negotiated  amount  based  on  historical  LMSC 
reproduction  experience.  Computer  Assisted  Design  and  Manufacturing  (CADAM) 
techniques  were  used  to  produce  160  technical  drawings.  Structural  stress  and  mo¬ 
ments  of  inertia  analysis  was  accomplished  using  a  UNIVAC  1 1 10  computer.  The 
estimated  computer  time  for  these  tasks  were  based  on  engineering  estimates.  The 
applied  rates  per  computer  hour  were  negotiated  and  based  upon  historical  LMSC 
data  processing  cost  experience.  Facilities  Capital  Cost  of  Money  (FCCM)  was  ap¬ 
plied  to  direct  labor  hour  estimates  at  a  negotiated  rates  for  all  categories  of  indirect 
costs.  The  CSTV  project  team  felt  that  unscheduled  overtime  would  undoubtably 
occur  as  development  and  test  functions  were  consciously  extended  and  completed 
beyond  an  eight  hour  shift.  Overtime  premium  costs  associated  with  this  likelihood 
was  composed  of  two  parts;  estimated  overtime  hours  and  a  negotiated  rate  to  be  ap¬ 
plied  to  these  hours.  The  primary  basis  for  the  estimated  overtime  hours  was  the 
LMSC  experience  in  association  with  a  similar  past  project  combined  with  the  esti¬ 
mates  of  project  members. 

7.  General  and  Administrative  Expense:  A  G&A  expense  rate  was  applied  to  the  CSTV 
project's  total  estimated  labor  hours.  This  negotiated  rate  was  based  on  historical 
LMSC  experience. 

I.  CHAPTER  SUMMARY 

This  chapter  began  with  a  system  description  of  the  Control  System  Test  Vehicle  pro¬ 
ject.  Contractor  responsibilities  and  task  requirements  were  outlined  and  discussed  in  terms 
of  the  Statement  of 'Work  and  the  Work  Breakdown  Structure  documents.  Various  organi¬ 
zational  aspects  of  Lockheed,  LMSC,  and  the  CSTV  project  were  seen  as  assimilating 
management  policies  and  the  project  team  structure  of  technical  undertakings  successfully 
accomplished  previously  within  the  Lockheed  Corporation.  The  complex  and  comprehen¬ 
sive  LMSC  cost  estimating  system  and  the  techniques  utilized  were  presented  from  the  per¬ 
spective  of  generating  valid  and  accurate  cost  estimates.  The  concept  of  time  phased  cost 
estimates  and  "sanity"  checks  provided  the  reader  with  insights  into  how  various  cost  esti¬ 
mation  techniques  arc  applied  throughout  the  development  process.  The  final  section  of  this 
chapter  points  out  significant  techniques  and  considerations  used  in  the  development  of  cost 
estimates  for  the  CSTV  project. 


IV.  PRESENTATION  AND  ANALYSIS  OF  THE  DATA 

A.  INTRODUCTION 

The  preceding  chapter  introduced  the  process  and  techniques  by  which  LMSC  gener¬ 
ates  cost  estimates  for  its  state -of- the -an  extension  projects.  This  process  was  followed  by 
the  CSTV  project  team  with  very  few  exceptions.  The  purpose  of  this  chapter  is  to  present 
and  analyze  the  results  of  the  LMSC  cost  estimation  process  of  the  CSTV  project.  First,  the 
original  CSTV  project  cost  estimate  and  the  results  of  the  Best  and  Final  Offer  contracting 
process  are  compared.  The  rationale  for  the  revised  cost  estimates  is  then  presented.  Next, 
a  presentation  and  analysis  of  the  predicted  and  the  actual  development  costs  experienced  is 
conducted.  This  data  is  presented  in  chronological  order  as  was  experienced  throughout  the 
life  of  the  project  and  includes  justifications  for  cost  growih  and  variances  amounts.  Con¬ 
clusions  are  then  reached  concerning  the  principle  factor(s)  which  significantly  impacted 
CSTV  project  costs. 

B.  INITIAL  CSTV  PROJECT  COST  ESTIMATES 

Appendix  B  contains  the  Contract  Pricing  Proposal  (DD  form  633-4)  dated  July  21, 
1978.  This  document  is  the  result  of  the  initial  cost  estimation  effort  on  the  part  of  the 
CSTV  project  team.  As  part  of  normal  contracting  procedures,  a  negotiation  process  be¬ 
tween  representatives  of  LMSC  and  the  government  followed  submission  of  the  Contract 
Pricing  Proposal.  What  resulted  was  a  "Best  and  Final  Offer"  (BAFO)  proposal  submitted 
by  LMSC  on  October  20,  1978.  Appendix  C  contains  the  Contract  Pricing  Proposal  (DD 
Form  633-4  dated  October  18,  1978)  which  was  provided  as  an  enclosure  to  the  BAFO 
submission.  As  is  evident  upon  comparison  of  these  two  documents,  the  total  estimated 
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project  cost  was  reduced  by  a  substantial  sum  ^  .  It  is  imponant  to  realize  that  some  of  the 
difference  in  estimated  project  cost  was  due  to  changes  in  contract  requirements  as  re¬ 
quested  by  the  government.  The  following  summary  shows  how  the  total  contract  values 
changed  as  a  result  of  this  negotiation  phase. 

PRIOR  TO  BEST  AND  HNAL  OFFER: 

Initial  estimated  cost  $  2,276,503 

Fixed  fee  191.862 


Total  CPFF  contract  amount 


$  2,468,365 


AFTER  BEST  AND  FINAL  OFFER: 
Initial  estimated  cost 


$  1,659,545 


Fixed  fee 

Total  CPFF  contract  amount 
DIFFERENCE; 


$  (693,963) 


PERCENTAGE  REDUCTION; 


(39.11%) 


There  were  several  reasons  for  the  reduction  in  estimated  project  costs;  and  are  pro¬ 
vided  as  follows. 


Materials  and  subcontracted  items  were  reduced  by  $313,050  as  a  result  of ; 

(a)  A  change  in  the  prime  subcontractor  responsible  for  manufacturing  the 
pressure  hull.  This  change  (from  REMCO  to  Niles  Engineering)  resulted  in 
a  significant  price  reduction. 

(b)  An  increase  in  government  furnished  items  (GFI)  as  a  result  of  negotiated 
contract  changes. 


^  The  BAFO  figures,  as  will  be  seen,  were  not  the  final  negotiated  amounts.  They  are 
presented  here  to  provide  a  complete  accounting  of  the  changes  in  cost  estimates  which 
occurred  throughout  the  project. 
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(c)  Market  research  into  sources  of  continued  suppliers  allowed  for  substitution 
of  lower  per  unit  cost  components. 

2.  Direct  Labor 

As  a  result  of  reevaluating  the  engineering  process,  estimated  engineering  labor 
requirements  were  reduced  by  2,470  hours.  Small  economies  and  design  improvements 
were  effected  to  reduce  the  projected  number  of  assembly  drawings  required.  An  additional 
833  labor  hours  of  test  engineering  efforts  were  eliminated  for  being"non-essential"  in  na¬ 
ture.  Total  direct  labor  dollars  were  reduced  from  $  524,743  to  $434,850;  a  difference  of 
$89,893. 

3.  Manufacturing 

Manufacturing  labor  estimates  were  reduced  by  4,673  hours.  This  was  due  to 
numerous  changes  in  the  basic  development  contract.  Because  these  changes  were  brought 
on  by  the  government,  for  the  purpose  of  this  study  ,  they  are  considered  "contract  reduc¬ 
tions  "  and  should  not  be  treated  as  true  cost  variances  or  overruns. 

4.  Computer  Time  and  Reproduction 

The  expected  project  computer  time  was  reduced  by  992  hours;  from  $63,414  to 
$33,654,  a  difference  of  $29,760.  This  was  a  direct  result  of  the  revaluation  of  the  engi¬ 
neering  process  and  corresponding  reduction  in  the  number  of  projected  assembly  draw¬ 
ings  required.  The  expected  reproduction  expense  was  also  reduced  by  $1,220  for  the 
above  cited  reasons. 

C.  CSTV  PROJECT  COST  GROWTH  AND  VARIANCE  ANALYSIS 

As  a  result  of  further  contract  negotiations  after  LMSC  submitted  it's  BAFO,  an  esti¬ 
mated  project  cost  of  $1,682,819,  and  a  fee  or  profit  of  $1 16,566  was  agreed  upon.  A 
CPFF  contract  was  signed  on  29  December  1978  and  work  on  the  CSTV  project  began 
soon  afterwards. 
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Despite  the  thorough  pre-contract  award  estimation  efforts  expended,  a  myriad  of 
technical  and  scheduling  difficulties  were  experienced  throughout  the  development  of  the 
CSTV  project.  The  initial  estimated  cost  figure  of  $1,682,819  and  the  scheduled  project 
completion  time  was  formally  changed  a  total  of  six  times  between  August  24,  1979  and 
February  12,  1981.  A  combination  of  contract  modifications  and  cost  variances  resulted  in 
a  final  actual  project  cost  of  $3,979,838.  The  final  completion  date  of  the  CSTV  project 
was  March  30,  1982;  more  than  two  years  beyond  the  initial  January  1980  target  comple¬ 
tion  date.  All  cost  figures  are  nominal  amounts  measured  in  then-year  dollars.  This  along 
with  subsequent  data  was  obtained  by  examination  of  LMSC  documentation  generated  as  a 
consequence  of  actual  cost  growth  and  overruns  experienced  throughout  the  life  of  the 
project.  The  negotiation/approval  process  associated  with  Cost  Plus  Fixed  Fee  (CPFF) 
contractual  agreements,  necessitated  the  format  of  the  reports  from  which  the  researcher 
compiled  the  data.  Accumulated  costs  are  presented  to  the  government  for  acceptance.  The 
basis  for  these  changes  in  estimated  costs  are  justified  by  contract  modifications  and  ex¬ 
pected  cost  variances  (expected  because  LMSC  cannot  incur  additional  costs  prior  to  gov¬ 
ernment  review  and  approval)  ^  .  Thus  a  periodic  negotiation  process  ensues  between  the 
government  and  LMSC  with  allowable  costs  lesulting. 

Much  of  the  data  reviewed  provided  information  on  both  costs  and  fee  (profit).  The 
objectives  of  the  study  limits  its  relevance  to  analysis  of  cost  data  exclusively.  The  fol¬ 
lowing  table  provides  a  chronological  summary  of  the  cost  estimation  changes  actually  ex¬ 
perienced  throughout  the  life  of  the  CSTV  project. 


^  Actual  cumulative  costs  form  the  basis  for  the  total  estimated  project  cost  and  is 
compared  to  the  cumulative  approved  costs  to  determine  the  expected  variance  amount.  See 
Appendix  D. 
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FINAL  NEGOTIATED  CONTRACT  GROWTH  &  VARIANCE  RESULTS 


\ 

1 


DATE 

ACTION 

MODinCATION 

VARIANCE 

TOTAL 

29  Dec  78 

Start  Date 

N/A 

N/A 

24  Aug  79 

Analysis  #1 

82,925 

524,798 

607,723 

12  Nov  79 

Analysis  #2 

192,986 

375,773 

568.759 

29  Feb  80 

Analysis  #3 

128,876 

345,014 

473,890 

24  Jun  80 

Analysis  #4 

239,817 

209,970 

449,787 

29  Aug  80 

Analysis  #5 

0 

90,510 

90,510 

10  Dec  80 

Modification* 

75,000 

0 

75,000 

12  Feb  81 

Analysis  #6 

0 

31,350 

31,350 

30  Mar  87 

Completion 

$719,604 

$1,577,415 

$2,297,019 

*  On  this  date,  it  was  agreed  upon  to  modify  the  contract  by  decreasing  the  fee  by 
S75,000  and  increasing  the  cost  by  the  same  amount. 


Detailed  data  (by  the  above  actual  reponing  dates)  concerning  CSTV  project  cost 
growth  and  variance  is  provided  in  Appendix  D  and  suppon  all  the  data  presented  in  the 
following  pages.  Reasons  for  these  difference  are  wonhy  of  examination  so  that  insights 
into  the  risks,  uncertainties,  and  technical  difficulties  associated  with  this  particular  state-of- 
the-art  extension  project  may  be  made.  Contract  modifications  as  requested  by  the  govern¬ 
ment  will  be  identified  but  not  be  discussed  in  length  as  these  estimated  costs  were  devel¬ 
oped  by  means  of  the  l,MSC  cost  estimation  process  and  models  previously  discussed.  In¬ 
sights  into  the  reasons  for  the  cost  variances  will  be  the  overriding  objective. The  primary 
rationale  for  each  revisions  of  the  cost  estimate  will  now  be  discussed  in  the  order  in  which 
they  occurred  throughout  the  life  of  the  CSTV  project. 


The  first  formal  change  in  the  CSTV  cost  estimate  occurred  on  August  24,  1979. 
The  cost  revision  as  of  this  date  is  summarized  as  follows: 


Cost 

Basic  Contract 

$1,682,819 

Contract  Modifications 

82,925 

Variance  or  Cost  Overmn 

524,798 

Revised  Estimated  Cost 

$2,290,542 

Estimated  F*roject  Completion  Date 

May  1980 

The  contract  was  modified  at  the  governments  request  to  include  1)  the  design 
and  manufacture  of  a  special  boom  assembly  ,  2)  the  procurement/design  of  special  filters, 
and  3)  an  increase  in  the  accuracy  of  a  specific  subsystem.  The  primary  reason  for  the  vari¬ 
ance  experienced  as  of  this  date  was  directly  contributable  to  the  complexity  and 
sophistication  of  the  CSTV  itself  In  197"  LMSC  was  awarded  a  preliminary  design  con¬ 
tract.  Design  studies  then  began  in  anticipation  of  a  Navy  RET.  Because  of  the  extensive 
preliminary  design  work  completed  prior  to  the  RET,  it  was  the  opinion  of  LMSC  that  very 
little  preliminary  design  would  be  required  at  the  time  that  the  contract  would  be  let.  As  a 
consequence,  the  LMSC  Best  and  Final  Offer  reflected  only  a  minor  amount  of  anticipated 
preliminary  design  costs.  However,  it  soon  became  apparent  that  Navy  specifications,  es¬ 
pecially  those  dealing  with  the  tail  section  design  and  other  technical  matters,  were  not  be¬ 
ing  precisely  meet  with  the  design  efforts  to  date.  Further  preliminary  design  work  was 
needed.  Further  problems  surfaced  once  actual  developmental  work  began.  The  complexity 
of  the  project  required  additional  drawings  (  from  136  to  321)  and  engineering  effort  in 
order  to  adequately  define  the  product.  In  addition,  a  number  of  procurement  difficulties 
were  experienced.  The  most  significant  being  the  rejection  of  unsatisfactory  hull  material 
from  Kaiser  Aluminum.  Reprocurement  of  this  material  amounted  to  project  delays  and 
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fruitless  efforts  at  obtaining  an  alternate  source  of  supply.  Finally,  significant  quantities  of 
highly  specialized  precision  components,  required  by  the  CSTV,  were  fabricated  by  LMSC 
when  they  proved  to  be  unavailable  "off-the-shelf  as  had  been  previously  expected.  In 
summary,  the  following  factors  contributed  to  the  cost  variance  at  this  stage  of  project  de¬ 
velopment: 

(a)  Substantially  more  preliminary  design  effon  than  anticipated. 

(b)  Significant  procurement  difficulties  experienced. 

(c)  Off-the-shelf  components  not  available  to  the  extent  planned. 

2.  CSTV  Cost  Growth  and  Variance  Analysis  Number  Two 

The  second  formal  change  in  the  CSTV  cost  estimate  occurred  on  November  12, 
1979.  The  cost  revision  as  of  this  date  is  summarized  as  follows: 


Cost 

Allowable  Costs  to  Date  $2,34 1,255 

Contract  Modifications  192,986 

Variance  or  Cost  Overrun  375.773 

Revised  Estimated  Cost  $2^910^014 

Estimated  Project  Completion  Date  May  1980 


The  contract  was  modified  at  the  governments  request  to  include  1)  funher  ef¬ 
forts  to  design  test  and  manufacture  an  additional  tail  section  for  the  CSTV  and  2)  an  up¬ 
graded  electronics  package.  Since  the  first  cost  growth  and  variance  analysis,  a  number  of 
technical  and  schedule  problems  developed  which  directly  impacted  the  estimated  project 
completion  costs  as  seen  above.  The  first  problem  had  to  do  with  CSTV  hull  fabrication 
difficulties  experienced  at  Niles  Machine  Inc.,  the  prime  subcontractor.  The  time  to  com¬ 
plete  welding  of  inserts,  closures,  and  foundations  was  greater  than  anticipated.  Significant 
numbers  of  LMSC  technicians  were  sent  to  the  subcontractor  site  in  an  attempt  to  rectify  the 
holdups.  In  the  end  it  became  necessary  to  transfer  entire  sections  of  the  hull  to  LMSC  for 
completion  of  welding  work.  Problems  with  the  control  electronics  and  procurement  of  a 
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critical  ballast  pump  assembly  was  the  second  difficulty  encountered  by  LMSC  at  this  time. 
Electrical  wiring  changes  were  necessary  after  initial  hull  sections  were  received  and  found 
to  be  more  densely  packed  than  anticipated.  Finally  the  overall  complexity  and  cumulative 
design  changes  thus  far  in  the  project  resulted  in  additional  quality  assurance,  rework,  and 
continued  engineering  design  efforts.  In  summary,  the  following  factors  contributed  di¬ 
rectly  to  the  continued  variances  in  development  costs: 

(a)  Hull  fabrication  difficulties. 

(b)  Procurement  difficulties  experienced  with  major  subassemblies. 

(c)  Technical  problems  associated  with  wiring  configuration  greater  than  anticipated. 

(d)  Additional  design  work  Q/A,  and  rework  beyond  expectations. 


The  third  formal  change  in  the  CSTV  cost  estimate  occurred  on  February  29, 
1980.  The  cost  revision  as  of  this  date  is  summarized  as  follows: 


Allowable  Costs  to  Date 
Contract  Modifications 
Variance  or  Cost  Overrun 
Revised  Estimated  Cost 
Estimated  Project  Completion  Date 


Cost 

$2,  859,301 
128,876 
345.014 
$3,333.191 
September  1980 


No  additional  contract  modifications  were  identified  as  of  this  reporting  date. 
However,  a  significant  cost  variance  was  experienced.  Additionally,  the  CSTV  project  in¬ 
curred  its  first  estimated  program  schedule  extension.  LMSC  was  experiencing  several 
unanticipated  procurement  and  subcontractor  performance  difficulties.  Delays  in  manufac¬ 
turing  critical  path  items,  such  as  the  sail  and  tail  subassemblies,  were  traced  to  the  lack  of 
required  component  pans.  These  events  not  only  resulted  in  project  delays  but  caused 
LMSC  to  expend  considerable  effons  in  resolving  these  problems.  Cost  overruns  not  di¬ 
rectly  linked  to  project  delays  due  to  lack  of  components,  primarily  included  continued 
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difficulties  with  hull  fabrication  and  unexpected  hardware  integration  problems  between 
off-the-shelf  components  (also  of  which  required  custom-made  connector  parts).  This 
situation  required  the  addition  of  more  engineering  and  administrative  man-hours  than 
expected.  Planned  expenditures  for  material  costs  were  not  significant  affected.  A  summary 

of  the  primary  reasons  for  the  increased  development  costs  estimate  include: 

(a)  Delays  associated  with  deliveries  of  component  parts;  some  of  which  directly  affect 
tasks  on  the  critical  path. 

(b)  Difficulties  with  connections  between  components 

(c)  Continued  problems  with  hull  fabrication 


The  fourth  formal  change  in  the  CSTV  cost  estimate  occurred  on  June  24,  1980. 


The  cost  revision  as  of  this  date  is  summarized  as  follows: 


Allowable  Costs  to  Date 


Contract  Modifications 
Variance  or  Cost  Overrun 
Revised  Estimated  Cost 
Estimated  Project  Completion  Date 


Cost 

$3,333,191 

239,817 

..-2Q9.97.Q 

$3,782.978 

February  1981 


The  contract  was  further  modified  at  the  government's  request  to  include  1)  a 
second  tail  assembly,  and  2)  a  modification  to  the  existing  installed  tape  recording  system. 
The  cost  variances  experienced  at  this  time  primarily  resulted  from  continuing  problems 
remaining  as  of  the  last  report  date.  These  difficulties  were  expected  to  be  resolved  rela¬ 
tively  quickly.  As  it  turned  out  however,  LMSC  was  unable  to  accomplish  this  without  uti¬ 
lizing  significant  additional  resources.  The  cost  overrun  was  primarily  attributed  to  contin¬ 
uous  subcontractor  difficulties  which  required  additional  LMSC  personnel  being  sent  to  the 
subcontractor's  sites  in  an  attempt  to  rectify  component  design  and  manufacturing  prob- 
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CSTV  Variance  Analysis  Number  Five 

The  fifth  formal  change  in  the  CSTV  cost  estimate  occurred  on  August  29,1980. 


The  cost  revision  as  of  this  date  is  summarized  as  follows: 

Cost 

Allowable  Costs  to  Date  $3,782,978 

Variance  or  Cost  Overrun  90.510 

Revised  Estimated  Cost  §3^873^4^ 

Estimated  Project  Completion  Date  January  1981 

No  additional  requests  for  contract  modifications  was  reported  at  the  time  of  this 
cost  analysis  report.  In  fact,  most  of  the  problems  experienced  earlier  seemed  to  be  re¬ 
solved.  The  variance,  as  verified  above,  was  the  smallest  anticipated  thus  far  in  the  CSTV 
project  life.  However,  a  relatively  new  technical  difficulty  surfaced  which  caused  the  con¬ 
tinued  high  usage  of  certain  engineering  and  manufacturing  personnel.  This  involved  re¬ 
pair,  modification,  and  isolated  testing  of  key  components  of  the  CSTV  control  mecha¬ 
nism.  As  will  be  seen,  this  problem  was  not  completely  resolved  prior  to  the  submission  of 
the  final  variance  report. 

6.  CSTV  Variance  Analysis  Number  Six 

The  final  formal  change  in  the  CSTV  cost  estimate  occurred  on  February 
12,1981 .  The  cost  revision  as  of  this  date  is  summarized  as  follows: 

Cost 

Allowable  Costs  to  Date  $3,948,488  * 

Variance  or  Cost  Overrun  31.35Q 

Revised  Estimated  Cost  $3^979^838  (Final  Cost) 

Actual  Project  Completion  Date  March  1981 

*  On  10  Dec.  80,  it  was  agreed  upon  to  further  modify  the  contract  by  increasing 
allowable  project  costs  by  575,000. 

No  further  government  induced  contract  modifications  were  experienced  through 
the  completion  of  the  CSTV  project.  The  anticipated  variance  was  attributed  to  remaining 


technical  problems  associated  with  the  CSTV  control  system.  In  addition,  the  costs  for  the 
second  tail  assembly  and  computer  graphics  time  for  final  drawings  were  greater  than  ex¬ 
pected. 

The  objective  of  the  preceding  paragraphs  was  to  present  the  estimated  and  actual  cost 
data  in  a  format  that  would  be  easily  comprehended  by  the  reader.  Furthermore,  the  reasons 
for  the  increase  in  project  costs  were  provided  so  as  to  identify  the  various  factors  impact¬ 
ing  on  cost.  As  is  apparent  by  observing  the  differences  between  the  beginning  cost  esti¬ 
mate  and  the  actual  project  cost  results,  these  differences  are  significant.  Even  with  a  pro¬ 
cess  as  complete  and  exhaustively  detailed  as  the  LMSC  approach,  the  process  of  cost  es¬ 
timation  in  this  particular  case  was  an  exceedingly  challenging  one. 

In  the  opinion  of  the  researcher,  the  value  of  a  model  such  as  the  LMSC  estimating 
system  stems  not  only  from  its  capacity  to  accurately  predict  future  costs  but  also  in  its 
ability  to  enhance  the  identification  of  factors  influencing  it's  outcome.  It  was  shown  that 
the  total  difference  is  a  combination  of  the  CSTV  project’s  cost  growth  and  cost  variance  or 
overruns.  It  would  appear  that  several  factors  throughout  the  development  of  the  CSTV 
project  contributed  significantly  to  increases  in  actual  cosL  The  degree  of  technological  ex¬ 
tension  required  would  be  expected  to  directly  affect  development  cost  by  increasing  un¬ 
certainty  associated  with  various  task  accomplishment.  The  CSTV  project  costs  appeared  to 
be  adversely  affected  by  this  task  uncertainty.  In  fact,  by  observing  the  data  contained  in 
Table  III,  it  is  interesting  to  note  that  variance  amounts  appear  to  be  greatest  in  the  initial 
stages  of  project  development.  Moreover,  these  amounts  decrease  with  the  life  of  the  pro¬ 
ject  and  presumably  with  a  reduction  in  task  uncertainty.  It  was  seen  that  extensive 
difficulties  were  initially  experienced  and  that  extended  preliminary  design  work  was 
required.  The  uncertainty  factor  seemed  to  shift  away  from  LMSC  personnel  and  more  to¬ 
wards  subcontractors.  As  was  observed,  the  major  reason  for  cost  overruns  in  the  later 


stages  of  development  appeared  to  be  design  and  manufacturing  difficulty  with  subcon¬ 
tractors.  Numerous  LMSC  technical  visits  were  conducted  to  assist  subcontractors  in  suc¬ 
cessfully  achieving  key  project  tasks.  Also  apparent  from  the  data  included  in  Table  III  is 
the  observation  that  contract  modifications  occurred  more  towards  the  beginning  and  mid¬ 
dle  phases  of  the  project  than  at  the  end.  This  may  be  a  function  of  product  definition  and 
clarity  as  more  tangible  results  are  realized.  To  the  extent  that  contract  modifications  affect 
variance  amounts  is  less  conclusive  from  observation  of  the  data.  A  final  factor  that  ap¬ 
peared  to  adversely  affect  development  costs  in  this  specific  case  is  that  of  material  delays 
of  singularly  less  critical  off-the-shelf  components  and  their  successful  integration  with 
other  component  parts.  These  difficulties  caused  severe  delays  in  the  project,  unproductive 
efforts  to  find  alternative  sources  of  supplies,  and  excessive  utilization  of  expensive  engi¬ 
neering  labor  in  redesign  efforts. 

D.  CHAPTER  SUMMARY 

This  chapter  presented  and  analyzed  the  actual  and  estimated  development  costs 
associated  with  the  CSTV  project.  Data  were  presented  in  chronological  order  as  they  were 
experienced  throughout  the  life  of  the  project.  Modifications  to  the  contract,  as  requested  by 
the  government,  were  identified.  However  primary  attention  was  paid  to  the  cost  overruns. 
Reasons  were  then  given  for  the  resulting  cost  variances.  Finally,  several  conclusions  con¬ 
cerning  factors  which  appeared  to  significantly  affect  development  costs  where  reached. 


V.  PRINCIPLE  FINDINGS,  CONCLUSIONS,  AND 
RECOMMENDATIONS 


A.  PRINCIPLE  FINDINGS  AND  CONCLUSIONS 

The  primary  objective  of  this  study  was  to  determine  what  methods  were  used  in  esti¬ 
mating  the  development  costs  associated  with  an  actual  state-of-the-art  extension  project. 
Lockheed  Missiles  and  Space  Company,  Inc.  and  the  development  of  the  CSTV  project 
provided  the  basis  for  this  inquiry.  The  principle  findings  and  conclusions  were  derived 
from  personal  interviews  with  key  LMSC  management  personnel  and  relevant  documenta¬ 
tion  collected  and  analyzed  at  LMSC  and  the  Lockheed  Naval  Plant  Representative  Office; 
both  located  in  Sunnyvale,  California.  These  findings  and  conclusions  are  presented  and 
discussed  as  follows. 


System,  was  utilized  to  predict  the  development  cost  of  the  CSTV  project.  This  process 
consists  of  a  series  of  events  within  phases.  Along  with  producing  creditable  cost  esti¬ 
mates,  the  estimating  process  was  used  to  assign  task  assignments  to  accountable  person¬ 
nel,  and  to  thoroughly  document  cost  estimates  for  internal  and  government  audit  purposes. 


though  the  Engineering  or  ”Bottom-Up”  cost  estimating  technique  was  predominantly  uti¬ 
lized  on  the  CSTV  project,  the  associated  program  method  was  relied  upon  to  a  lesser  ex¬ 
tent  when  program  comparisons  were  considered  appropriate.  Parametric  techniques  were 
used  in  the  final  cost  estimation  review  phase  to  provide  "sanity"  checks.  They  were  not 
relied  upon  in  the  primary  stages  of  generating  cost  estimates. 
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managerial  requirements,  including  tight  project  control,  and  the  establishment  of  a  single 
responsible  project  manager,  were  accomplished  in  this  manner.  This  project  team  organi¬ 
zational  structure  was  based  upon  the  successes  experienced  by  several  Lockheed  compa¬ 
nies  involved  with  similar  SOA  extension  projects  previously  undenaken. 

Despite  the  elaborate  nature  of  the  LMSC  cost  estimation  process  employed.,  sig¬ 
nificant  cost  growth  and  overruns  (variances)  resulted.  The  final  CSTV  project  cost  in¬ 
creased  dramatically  over  the  initial  estimated  costs.  This  led  the  researcher  to  conclude  that 
the  process  of  cost  estimation  utilized  by  LMSC  to  estimate  the  CSTV  project's  costs  was 
complicated  by  the  degree  of  technological  extension  required. 

Task  uncenainties  associated  with  the  development  of  the  CSTV  project  apDcaredlQ.  be 
the  major  contributor  to  the  cost  overruns  experienced.  Although  significant  preliminary 
design  work  was  conducted  prior  to  the  award  of  the  CSTV  contract,  numerous  additional 
engineering  labor  hours  were  incurred  as  project  engineers  attempted  to  adequately  define 
the  CSTV.  These  design  efforts  appeared  to  lessen  in  the  later  stages  of  project  develop¬ 
ment. 

B.  RECOMMENDATIONS 

1 .  Government  Officials  responsible  for  SOA  extension  projects  should  continue  to 
thoroughly  review  cost  estimates  submitted  by  contractors.  Emphasis  and  attention  should 
be  placed  on  whether,  or  to  what  extent,  the  contractor  fully  understands  the  requirements 
of  the  project. 

2.  Continue  to  insure  that  contract  documentation  such  as  the  SOW,  CDRL,  and 
WBS  for  state-of-the-art  extension  projects  is  clearly  defined  and  understood  by  all  con¬ 
cerned  parties.  This  may  help  to  minimize  the  possibility  of  project  misunderstandings  and 
underestimation  of  the  costs  associated  with  project  development. 
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3.  Move  towards  more  fixed  price  contracts  that  place  greater  risk  upon  the  contrac¬ 
tor.  SOA  extension  projects  appear  to  be  filled  with  numerous  uncertainties  caused 
primarily  by  the  degree  to  which  extended  technology  is  required.  The  process  of  estimat¬ 
ing  expected  costs  that  adequately  accounts  for  project  uncenainties  was  not  successfully 
realized  in  this  particular  case  and  resulted  in  the  government  incurring  significantly  more 
costs  than  anticipated.  This  also  may  be  the  experience  with  cost  estimation  process  em¬ 
ployed  in  other  projects. 

C.  RECOMMENDATIONS  FOR  FURTHER  RESEARCH 

If  total  contract  cost  growth  is  used  as  a  measure  of  effectiveness,  it  is  apparent  that  the 
process  utilized  in  estimating  the  final  project  cost  was  not,  for  various  reasons,  com¬ 
pletely  successful  in  the  case  of  the  CSTV  project-  This  study  involved  one  specific  case 
and  cannot  reasonably  be  expected  to  fully  support  any  broad  based  theoretical  conclusions 
on  its  own.  It  is  possible  but  unlikely  that  the  cost  growth  and  overruns  are  unique  to  this 
panicular  case.  Therefore,  it  is  recommended  that  further  case  studies  involving  SOA  ex¬ 
tension  projects  be  conducted  to  determine  if  similar  difficulties  are  encountered.  The  focus 
of  these  additional  case  studies  should  be  to  provide  further  identification  of  the  factors  af¬ 
fecting  cost  estimates  and  the  means  by  which  the  degree  of  required  technological  exten¬ 
sion  is  incorporated  into  the  cost  estimation  process.  Greater  insights  into  the  process  in¬ 
volved  would  assist  Navy  budget  analysts  and  program  managers  in  more  accurately  de¬ 
termining  cost  requirements  and  impacts  of  SOA  extension  projects. 

D.  FINAL  SUMMARY 

As  a  final  summary  of  the  information  presented  and  discussed  in  this  report,  the  pri¬ 
mary  and  subsidiary  research  questions  will  be  reinstated  and  briefly  answered. 
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*  Primary  Research  Question: 

What  methods  were  employed  by  a  major  Department  of  the  Navy  contractor  to  predict 
the  development  costs  associated  with  an  actual  state-of-the-art  (SOA)  extension 
project? 

Answer;  A  combination  of  the  engineering  (  primary  technique),  associated  program, 
and  parametric  (used  by  LMSC  cost  estimation  reviewing  authorities)  techniques 
within  the  framework  of  the  LMSC  Estimating  System. 

Subsidiary  Research  Questions: 

1 .  What  were  the  expected  costs,  actual  costs,  and  variances  experienced  during  the 
development  of  the  SOA  extension  project? 

Answer:  Details  on  costs  and  variances  are  provided  in  Chapter  IV  and  in  Appendix  D. 
Significant  cost  variances  and  contract  growth  resulted. 

2.  What  were  the  reasons  for  the  variances  experienced? 

Answer:  The  primary  reason  stems  from  the  costs  associated  with  the  engineering  re 
design  efforts  required  as  a  consequence  of  the  uncertainty  associated  with  the 
advanced  and  complex  technological  nature  of  the  CSTV  project.  Additionally,  an 
incomplete  understanding  of  contract  requirements  lead  to  unexpected  cost  increases. 

3.  Which  development  cost  categories  had  the  most  effect  on  total  project  cost?,  i.e.  what 
were  the  "cost  drivers"? 

Answer:  In  the  early  stages  of  the  project,  the  engineering  labor  cost  to  adequately 
define  the  CSTV  product  was  the  cost  category  most  affecting  the  project  cost.  In  the 
later  stages,  costs  were  associated  with  subcontractor  performance. 


4.  What  organizational  design  or  structure  was  utilized  during  the  SOA  project;  and  how 


does  that  design  affect  cost  management? 

Answer:  A  project  team  approach  was  utilized;  the  design  of  which  was  adapted  from 
successful  SOA  extension  projects  previously  undertaken  by  other  organizations 
within  the  Corporation.  Concern  over  project  costs  received  a  greater  degree  of 
management  attention  than  would  have  occurred  if  more  traditional  organizational 
approaches  had  been  adopted. 
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APPENDIX  A 

CSTV  STATEMENT  OF  WORK 


The  Statement  of  Work  describes  the  tasks  to  be  preformed  by  LMSC  in  the  design,  con¬ 
struction,  and  test  of  the  Control  System  Test  Vehicle  (CSTV)  and  support  equipment. 
This  information  was  reproduced  from  the  LMSC  management  proposal  for  the  CSTV 
project  submitted  to  NAVSEA  on  25  July  1978. 

SCOPE 

The  contractor  shall  make  maximum  use  of  all  data  developed  and  documented  under  Con¬ 
tract  N0C)04-78-C-5309.  All  work  to  be  preformed  under  this  contract  is  generally  de¬ 
scribed  below: 


ITEM 

0001-  Design,  construct,  test  and  deliver  one  (1)  Control  System  Test  Vehicle  and  one  (1) 
set  of  support  equipment  as  defined  herein. 

0002-  Provide  interim  repair  parts,  supplies  and  services  as  ordered  by  the  Administrative 
Contracting  Officer  (ACO)  and  authorized  by  contract  modification. 

0003-  Prepare  and  deliver  as  defined  on  the  Contract  Data  Requirements  List. 

PERIOD  OF  PERFORMANCE 

The  contractor's  effort  described  in  this  Statement  of  Work  shall  be  accomplished  during 
the  periods  specified  below; 

ITEM 

0001-  From  contract  award  through  the  twelfth  (12)  month  after  contract  award. 


0002-  As  specified  in  the  authorizing  contract  modification. 

0003-  From  contract  award  through  the  fourteenth  (14)  month  after  contract  award. 


APPLICABLE  DOCUMENTS 

The  following  documents  of  the  issues  and  date  specified  form  a  part  of  this  SOW  to  the 
extent  specified  herein. 

MILITARY 

NO-794-78-003  April  1978  "Control  System  Test  Vehicle 

and  Equipment  Specification" 

Exhibit  A  to  10  May  1978  ASCOP  Test  Vehicle  Contract 

N(XX)24-78-PR-31018  Data  Requirements 
List  (DD  Form  1423) 


TASK  DESCRIPTION 

This  section  defined  the  tasks  to  be  performed  by  the  contractor  and  the  responsibility 
interfaces  between  the  contractor,  NAVSEA,  and  other  participating  activities.  Paragraph 
numbers  herein  are  directly  relatable  to  the  Work  Breakdown  Structure  (WBS)  included  in 
the  Control  System  Test  Vehicle  Specification,  NO-794-78-003  and  the  contractor's  Man¬ 
agement  Proposal. 

Model  Vehicle.  Design  and  construct  one  Test  Vehicle  in  accordance  with  the  requirements 
of  Specification  NO-794-78-003  and  Appendixes  1  through  5  thereof 

•  Perform  Design  Studies,  Design  Analyses,  and  Producibility  Studies  to  support 
the  detail  design. 


66 


•  Integrate  GFE  into  the  CSTY  design,  with  specific  emphasis  on  the  Inertial 
Measurement  Unit  (EMU)  and  the  Control  and  Recording  System  (CRS). 

•  Generate  detail  design  drawings. 

•  Prepare  a  design  Analysis  Report  (CDRL  Sequence  A004). 

•  Prepare  Critical  Design  Review  Data  (CDRL  Sequence  A003),  and  support  the 
Critical  Design  Review. 

•  Support  internal  design  reviews. 

•  Provide  technical  direction  to  subcontractors. 


•  Support  internal  design  reviews. 

•  Provide  technical  direction  to  suppliers  of  purchased  equipment. 

•  Manufacture,  inspect,  assemble,  and  test  the  support  equipment. 

•  Provide  interface  information  as  required  to  facilitate  Government  modification  of 
the  GFE  trailer  and  sled. 


Test  and  Evaluation.  The  contractor  shall  conduct  a  test  and  evaluation  program  to  verify 
that  the  vehicle,  including  contractor  furnished  equipment  and  Government  furnished 
equipment,  performs  in  accordance  with  the  requirements  of  Specification  NO-794-78-003. 

•  Conduct  development  tests  of  selected  items  as  required  to  evaluate  performance 
and  reliability. 

•  Prepare  a  factory  acceptance  test  plan  covering  factory-level  tests  and  special  tests 
for  Navy  approval  (CDRL  Sequence  AOOA). 

•  Conduct  factory  acceptance  tests  in  accordance  with  the  approved  test  plan, 
analyze  test  data,  and  prepare  test  reports  (CDRL  Sequence  A(X)B). 

•  Support  special  tests  at  the  Ocean  Simulation  Facility,  Panama  City,  FL,  analyze 
test  data,  and  prepare  special  report  (CDRL  AOOC). 

System  Engineering.  This  task  covers  the  integrating  activities  related  to  applied  engineer¬ 
ing  disciplines.  This  activity  will: 

•  Coordinate  with  NAVSEA  and  GFE  suppliers  to  define  requirements  and 
establish  physical  and  functional  interfaces  between  the  vehicle,  support 
equipment,  and  the  related  GFE. 

•  Maintain  coordination  with  NAVSEA  to  develop  software  interfaces  for  the  GFE 
IMU  to  assure  successful  operation  of  the  CSTV. 


•  Maintain  positive  interface  control  between  the  vehicle  and  support  equipment. 

•  Conduct  internal  design  reviews. 

•  Prepare  the  Critical  Design  Review  Agenda  (CDRL  Sequence  A(X)3),  CDR  Data 
(CDRL  Sequence  A(X)2),  and  conduct  the  CDR. 

•  Review  Test  Plans  and  Test  Reports. 

•  Prepare  for  Navy  approval,  a  Quality  Assurance  Plan  (CDRL  Sequence  AOOE) 
and  conduct  a  quality  assurance  program  in  accordance  therewith. 

•  Conduct  and  monitor  Reliability  and  Maintainability  programs. 

•  Conduct  and  monitor  Systems  Safety  and  Human  Engineering  programs. 

•  Prepare  requests  for  approval  of  nonstandard  parts  (CDRL  Sequence  A(X)7). 

•  Prepare  Quarterly  Technical  Progress  Reports  (CDRL  Sequence  A009)  and  a 
Final  Engineering  Repon. 

•  Construct  full  scale  soft  mock-ups  of  the  test  vehicle  and  support  equipment,  as 
required. 


Systems  Management.  The  contractor  shall  provide  and  maintain  an  organization  structure 
to  ensure  effective  direction  and  management  that  will; 

•  Formulate  decisions  and  exercise  technical  management. 

•  Establish  rational  priorities  and  clearly  defined  responsibilities  within  each 
discipline. 

•  Monitor  and  evaluate  activities  of  all  disciplines  to  ensure  feedback  of  complete 
and  accurate  information  for  in-depth  program  visibility, 

•  Establish  and  maintain  a  program  master  schedule  encompassing  all  tasks 
necessary  for  program  accomplishment. 
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•  Negotiate  resource  allocations  with  all  organizations,  document  results,  and 
release  authorizing  documents  for  resource  utilization. 

•  Coordinate  the  evaluation  and  selection  of  qualified  subcontractors  and  suppliers. 

•  Administer  all  subcontract  and  purchase  effon. 

•  Prepare  Monthly  Letter  Status  Reports  (CDRL  Sequence  AOOl). 

•  Ensure  thorough  evaluation  and  proper  authorization  of  all  design  changes. 

•  Confirm  the  incorporation  of  all  authorized  changes  into  hardware. 

•  Prepare  Engineering  Change  Proposals,  Deviations,  and  Waivers  (CDRL 
Sequence  AOOS). 

•  Ensure  the  proper  preparation  and  timely  submission  of  all  data  specified  by  the 
Contract  Data  Requirements  List  (DD  Form  1423). 

•  Perform  logistical  studies  and  analyses. 

•  Prepare  a  list  of  recommended  spare  parts. 

•  Prepare  a  list  of  special  and  general  purpose  electronic  test  equipment  (CDRL 
Sequence  AOOS). 

•  Prepare  a  Technical  Maintenance  Manual,  Operator's  Manual  and  Parts  List 
(CDRL  Sequence  A006). 

•  Perform  liaison  with  the  government  and  GFE  suppliers  to  ensure  integration  of 
GFE  Operation  and  Maintenance  Manuals. 

•  Inspect  and  test  GFE  as  received  to  ensure  proper  operation  before  installation. 

•  Provide  protection  and  control  of  GFE  to  prevent  damage  during  handling  and 
storage. 

•  Provide  field  support  as  directed  by  the  ACO  under  Item  0002  of  the  contract. 
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fiCOTAd  dbiwoiMdnf.  ddfdhi  hwhidorA.  pofoni  ipplicdiion  Aondf 
mpwbdrd.  dr  dlhor  bdAtt  am  which  dto  rordftr  ia  PArAd'o.  Ariof 
doocrrpVdh,  inclMdinb  amt  port  dr  owdol  nwdiddrA  of  ooed  cdiv 
•rdcf  iionr  or  cohpprvohf  on  which  dio  ivroffr  ro  porodfw  Pdh 
cOTiod#  df  ddflor  fdfo  df  modflr  Por  vrMf.  whif  pricd  df  cdrhrdcf 
iidd^  MtpdddT  df  tpvid  pid  rdfdf  dofior  omdwnf  of  rorofliop 
Ih  dbdtiidrv  if  9dciffcdffr  rvprdofod  At  (ho  cdhirociih#  offfcoo 
d  CApr  of  Ihd  cwTVhi  ficdhdd  dbridhprn  ptd  fddhiilicdridd  dl 
^•fiCdPfd  cldAAAd  dl  dPdcKic  pdldrNd  Ahdff  bd  pPdfTbdA 

J/  PrvMtfd  d  ffdf  df  phhcfVdl  fiddid  wrfhfn  ddcb  cpdbdrf  fir 
dicdfiiid  AndOTA  dr  dhficipPdd  ddwrcd.  piAniiir.  tpwr  phc^ 
cAMpdiiridM  dAfdihdC  PAd  bdord  df  OAidbliAhidb  owwrc a  Ahd 
rvdAOTAPidhddd  dl  odoA. 
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•«*«o««ico  *•«  •«>«•«•  ««  YOU*  4Crau«^t  am  •ccd«e»  ••  co»«<cr>o«  •*t»  *•* 

COWAACT  0«  »W«C0«T««CT  •ir«4«  ?««  W€w«€  ••«»«•* 

^  •••  ^  ~  j  Connnuinc  Audit 


CCAA.  P.O.  3<rt  J04.  Sunawaie.  CA  «40«< 


f40S^  T42-5991 


90  «Ou  •«Owi«<  ftO«Caai«i««r  CO«r*«CT  *i«AwC««C  TO 


«tt  ^  <fl  «««. 


. . BilUagi 


«.  9Q  *Ou  ao*  "Oca  *«▼  79«*a«C^  •• 

cawhta  «o«  av  ▼««»  ••o*o«to  coa^aAcr 


ca««o««  •4t«  r«c  co«r  ••<aci*w-t  act  #o»t«  •«  «t»o.  »ceT«OM  ■«  («••  a  ««•  tiu 


tHSTRUCTtOMi  T*) 

1.  T>m  •«»«•••  •<  fcMa  la  i«  ^aaiOa  •  ataMaM  lanaai  Of  aiaaa^taa  rat 

•Mtaa  (Na  atta^*  «aa««ia  ta  t»a  Gapamwaaw  a  aaaMaaa  m4  y»>  %t»a«  attaaaa 

aaavaa  a««a  aaita«i4aa  mmm%  t  mtmmmmrn  *•  tmmmmittm 

aatiaata  <ar  aataitaa  wa«a«  aaa  anataaia.  ta  laa  ••##«  aaaattaaa 

at  a  caMfaat  r«aatiM»c  '*«■  tata  aaaaaaai  tt«a  atlava*  aaait,  t#  «t«a  Ca«M«a 

«Maa  taa  caMiitiaaa  ataiaa  la  i«MT  J.  aa  raawaa  ta 

attaaiit  •  CaMtIicata  mi  Qmrmm  Caa<  m  ^WMit  Data  («aa  d$Pt^  .  - - 


aiMaa«taa  ratatia4  ta  taa  taa*  aiasaaa  «*a  mm  laMO* 

%aaa  a«ta«Maaai  a4  aaaaa*«ia9  aaai  a*  aaiaiaf  aata  ta  t%a 
•a  tawaaaiiaaaia.  im  aata  otil  aa  aaaa»<»aattf  taavtitiaa  aao 
aaaattaaa  /a«ta  aaaaa»<a«  aa  •aa*aa»*a«a>.  aaa  mmmm  avaitaaia 
ta  taa  Caaawiattat  Olftaaa  •  a«a  raaaaaaaaaataa  aaaa  raavaac 


7.  Aa  aoM  mi  tna  aaoviOa  tataPMaMtaa  taoavai  aa  tma  fatal,  tma 
atfarv  a»aat  aaaaai  ^tta  tiaa  i«at.  aao  ciaaatv  <a#aatfa  aa  aata. 
aaat  m  attataf  aata  nmm»  «a.  aata  aHM<a  ta  v^traata  ana  laa* 
taat  mm  ataaawaa  •#  aat«M«a  «a  Ai#t9  )«  aaaittaa. 

aa  MMiat  laaaiit  •tin  taia  <a»«>  a^a  auartaattaa  raaaaaaata  ta» 
aaiaaa  «a  atatata  taa  atfatar  a  aattaMttaa  aaaaaaa.  laritaiwtf: 
a.  <»a  i«at"w<«ai  laatva  aaatM>a  aaa  tita  awtt^tMtMai 
at  at^  aaa«««aa  atwA  ai  taa  aaitaiata  laoMiaa  taaaa 
aaaa  4a  arataaiiao  ifaai  aiiM«  aata.  mm» 


*■  Tiia  faaaiata  (m  taa  *Caa«  tlaaMaaa*  aao  laa  **aavaaaO 
Caaoaat  KaiMaata*  aaa  aaa  wattaaa  aa  ttfra  mmmmmmmmmm^ 
*P^aa  aiaa  aa  mmrnmmmrn  m  aulanMM  laataat  atta  tita  vtaa  a^  I 
arairai  mt  taa  Cataaaaaan  Otttaaa  ii  taaaaa#  iwMpa  a«taaa>aa 
aaa  atitataaa  a^aaatatiaa.  la  all  ataaa  iaaaaata  taia  tar»  aal 
aa  aaaaiwaa  aao  aaataiaaa  atttaaaa  taaaaa. 


a.  <aa  aa 
attaa. 


1.  Atiaaa  aaaa^ta  aaaaa  if  ttaaaaaara  < 
«aM  taa  atiaawawai  at  aaiiaa  taa  a^ai 


atfa  4a  lAta  aal* 


S.  9a  aaataataa  mi  taia  vaaaaai  affaaaa.  »l  aaiaataO  far 
aatiaaiatkan.  yaaaa  ta  uta  Caaaraatiaa  OfiMaa.  aa  aia  -  iiaw 
laaa  raaraaaiMatiaa.  ina  tt^ia  ta  aaaaiiaa.  imm  laa  awaaaa  mi 
^mmUfmt  t»a  aaaa  v  a^aatf  aata  taaaattaa.  laaaa  aaaaa. 
rmmmmm  aaaaaattta  aaa  ataa#  taaaaatiaf  aata  «tM«a  aill  aaa* 
•tt  aaaawata  aaataatiaa  aa  aaaa  taat  aa  aatataf  aata.  aiaaf 
attw  taa  caa»aatatta«»a  aaa  arataaiiaaa  aaa#  txavavk  Tltta 
tifat  «aa  aa  aaa^iaao  laaaaaaaiiaa  aaa  aaa  aataatatiaaa 


POOTMOtli 


i/  Caiaa  ta  fAta  mmtmmm  ttmmm  aaaaaaaaa  aatf  paaaaaaAia  aaaaa 
•aaaA  ta  <a«  mi  imm  affapar  aatl  paaaaata  mm  mwmmwg 

tm  imm  afftaiaaa  mmmtmmmtmmrn  mi  taa  aaataaat.  vaaa  mmm  mt  taa 
caata  ta  tat#  aat««at  aaaa  ataaaaa  iaaa  laataaaa  t  a.#.,  aa  a 

aaatraat  ar  «a«ifa  araaal.  aaaaaf  a  taaa  oa  «i  attaaaatf 
MMOMta#  aaaa«fia.  Immmttim  atl  aataa  «ia  ta«ia<ara  aataaaa 
rntmmtm,  aiataiaaa.  ar  ar#«itaaitaaa  mtrnmm  m  mmmmmmt  mm^ 
$mi.  vatta  ara  ataiaaaa  a«  ataar  taaa  tAa  tataar  af  aaat  ta  taa 
a^fiaat  tfmm$mmmm  mm  mmmmmm  mmm  at  aava* 


2/  Immmmmm  t*a  raraa  aaaa  mmm  mmmmrnmm  aa  aavaartata  aaataaa 
fiv*.  VAaaa  m^mmmmmmrn  mmm  aaaa  raaaaaa  area  OaaataaaaM  taa* 
raaaatafraaa  at  lAa  aaa  af  farwa^  prtata#  rataa.  aaaartaa  taa 
aatara  af  taa  atraaataaf.  ^wataa  taa  taataao  at  aaa«aaa«taa 
«ta  aavttaartaa  at  raar  ■  uta  aaa  mmrnmmmm,  •mmimmm$  tmmt 

aaaa  ta  mmmmm  m  aaaaa  far  aaataaftaa  af  laa  raaaaaaataaaaa 


raaaa  ta  arttwaa  t#  «Mf  aaatfaAla  ta  taAfOtf  A  ta 
rttaaa  aaa^ata  aagaa  aa  aaaaaaara  «»a  faaaritr  ra 


••atarwwa*  caitMM  taa  attaatWaM  »a  aataa  tatataiaftM 
I  laa  I  tita^  taa  raaatfta  aaat  atawaat  awa  Aa  latat*  Hm 
ara  tatwar  ta  araaataoaat  aa^^r.  taa  aaaa  ar  vtaftt#  awa  awar 
•  a  aaaiaata.  aanatata  aaa  «awaf.  aaa  tA#  iw^^taia  taatara 
aaaa  ta  arataatta^  aaa  taa  aata  ta  taa  mmammtmm  mmmt  Aa  ttataa 
ta  mmiltmtmmt  aatatl  la  Mata  t»a  Caarraatta#  Offtcar  t#  rratwara 
taa  aaaaaaat.  ^a#  atMWta.  araM#a  taa  Aaata  aaaa  ta*  ^ait^ 
mmmmmmtm  raaa  •#  ar  vaaaar  aaatattaaa.  aaaa  rrttatataa.  » 
'••••••  ^taaat  taa  raaaaa  tar  ara  af  arataaaa  ratar  atNaa  a^ 

ttatt>r^«tr  aaaar* aaaaa  rataa  ttraaaaa  -rr—a  a 
•atar  f mmmmmm ^mmmu  ata.M  ar  taaftftaattaa  tar  aa  ia» 

»•  »••••  t^ttetaataa  aa«a  «•#  ratara  raaraaaaai. 

ata.t.  taMtrr  aaatrw  aar  aaattaAwtrtaa  «at«a  rra  ta* 
aiaaaO  tm  taa  aaaaaaaa  a*«*a.  twaa  aa  Mttataataa  rart*  at  ra* 
iaata  aafaaftra  «ara,  m  awaatoavao  taataitaat  Olfftaaftta^ 


1/  If  tmm  tarat  ^rafta  aaaa  mtmmmrn  mmrnm  tm  tm  mmmmmm  mi  l}J# 
arartaa  aa  a  raaarara  aaga  ta#  aa  00  ^ara*  *l}.  Aarrttr 
ffaaarfi  ta#  tattai^ag  atiataiaiti  aa  aaaa  aaa^ata  ttaai  a# 
raratir  ar  Itcavaa  tra.-  a^aa  a»a  aa^ata  af  'traaaart  aata  af 
ttaaaaa  a#ra«aaa»t  watt  awaaar*.  aa*ra*  aaottaattaa  rarta# 
ta^tOara,  v  atav  aaata  aa  ataaiA  laa  raratt*  la  aaraatat  at*af 
#aaar*atta^  mmm  aar*  ar  mtmrnmf  aMaoara  a*  aarA  taa> 

mmmt  itmm  mm  mmmmm»mm  mm  waica  taa  raratir  '*  aaraatat  mmm 
mmrnmmmm  mm  aattar  taia  at  taaaH?  aa*  ^tt  mmrnm  mi  tmmmmmrnt 

mmmm  aMOar  af  ^lat  mmm  tmmt  aattar  mmmmmi  mt  *arattiaab 
fa  aaaittati.  if  taaaittratir  ra«ta*iaa  ar  laa  «aMrav...4  ataaa* 
a  aaaa  at  ta#  aa*aaM  naati*a  aaraaaiaa*  aia  ia«Mifiaatiaa  af 
aaattaaat#  ata*aia  mi  aaaatfia  aa*«if*  aaati  aa  ara**aa* 


^^^aa  a  ttar  af  artaaiaat  itmmm  mtirnmm  raaa  caaaaara  aa» 
ataa*tf»a  a»aM  aa  ■wtttaavaa  ravraa.  aaaAtttr.  4Mtt  an*  a. 
caaiaavtittBi  aotattia*  ana  Aaata  at  aataattaAit^  aavaa  aaitf 


|!(j^  S^atloftlo  »o  DTIC  aoo*  not 

panit  lully  leo>bl«  lopioducttoa 
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